[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 478211338 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002243] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010130] LSM: Security Framework initializing [ 0.011053] Yama: becoming mindful. [ 0.012049] SELinux: Initializing. [ 0.013110] *** VALIDATE selinux *** [ 0.022138] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027214] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031099] *** VALIDATE tmpfs *** [ 0.033118] *** VALIDATE proc *** [ 0.034258] *** VALIDATE cgroup *** [ 0.035008] *** VALIDATE cgroup2 *** [ 0.036246] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039034] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.043524] debug: unmapping init [mem 0xffffffffab659000-0xffffffffab660fff] [ 0.045186] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046771] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047026] ... version: 2 [ 0.048014] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053017] ... event mask: 000000070000000f [ 0.054335] rcu: Hierarchical SRCU implementation. [ 0.056361] smp: Bringing up secondary CPUs ... [ 0.057647] x86: Booting SMP configuration: [ 0.058030] .... node #0, CPUs: #1 #2 #3 [ 0.061092] smp: Brought up 1 node, 4 CPUs [ 0.063013] smpboot: Max logical packages: 1 [ 0.064023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145451] node 0 deferred pages initialised in 79ms [ 0.149013] devtmpfs: initialized [ 0.150085] x86/mm: Memory block size: 128MB [ 0.152862] gcov: version magic: 0x41383552 [ 0.154258] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155085] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156263] pinctrl core: initialized pinctrl subsystem [ 0.157279] [ 0.157868] ************************************************************* [ 0.158013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159013] ** ** [ 0.160015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161020] ** ** [ 0.162014] ** This means that this kernel is built to expose internal ** [ 0.163017] ** IOMMU data structures, which may compromise security on ** [ 0.164015] ** your system. ** [ 0.165013] ** ** [ 0.166015] ** If you see this message and you are not debugging the ** [ 0.167017] ** kernel, report this immediately to your vendor! ** [ 0.168016] ** ** [ 0.169015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170012] ************************************************************* [ 0.171657] NET: Registered protocol family 16 [ 0.172476] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.173079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.174051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.175338] cpuidle: using governor menu [ 0.177874] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.179957] PCI: Using configuration type 1 for base access [ 0.182166] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.193069] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.194034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.195215] cryptd: max_cpu_qlen set to 1000 [ 0.197885] ACPI: Added _OSI(Module Device) [ 0.200022] ACPI: Added _OSI(Processor Device) [ 0.201013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.203017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.208158] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.215173] ACPI: Interpreter enabled [ 0.216064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.218018] ACPI: Using IOAPIC for interrupt routing [ 0.221105] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.225445] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.236989] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.239050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.242021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.245080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.249347] acpiphp: Slot [2] registered [ 0.251135] acpiphp: Slot [5] registered [ 0.253231] acpiphp: Slot [6] registered [ 0.255252] acpiphp: Slot [7] registered [ 0.257263] acpiphp: Slot [8] registered [ 0.259205] acpiphp: Slot [9] registered [ 0.261204] acpiphp: Slot [10] registered [ 0.263234] acpiphp: Slot [3] registered [ 0.264194] acpiphp: Slot [4] registered [ 0.266143] acpiphp: Slot [11] registered [ 0.268157] acpiphp: Slot [12] registered [ 0.269201] acpiphp: Slot [13] registered [ 0.271152] acpiphp: Slot [14] registered [ 0.273148] acpiphp: Slot [15] registered [ 0.274126] acpiphp: Slot [16] registered [ 0.276162] acpiphp: Slot [17] registered [ 0.277144] acpiphp: Slot [18] registered [ 0.279171] acpiphp: Slot [19] registered [ 0.281141] acpiphp: Slot [20] registered [ 0.282139] acpiphp: Slot [21] registered [ 0.284135] acpiphp: Slot [22] registered [ 0.286183] acpiphp: Slot [23] registered [ 0.287129] acpiphp: Slot [24] registered [ 0.289151] acpiphp: Slot [25] registered [ 0.291205] acpiphp: Slot [26] registered [ 0.292135] acpiphp: Slot [27] registered [ 0.294141] acpiphp: Slot [28] registered [ 0.296150] acpiphp: Slot [29] registered [ 0.297296] acpiphp: Slot [30] registered [ 0.299125] acpiphp: Slot [31] registered [ 0.300139] PCI host bridge to bus 0000:00 [ 0.302033] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.304035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.306031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.309041] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.312038] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.315054] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.317258] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.320776] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.324419] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.333827] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.340073] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.343025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.345022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.348029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.351746] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.354195] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.357046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.360965] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.366016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.382020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.387017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.393469] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.406023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.415023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.437018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.448273] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.457017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.463021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.482027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.497385] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.506024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.512024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.528019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.540754] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.547021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.556022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.572035] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.590132] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.600024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.608070] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.628026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.641435] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.649022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.655018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.674028] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.688660] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.692044] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.694435] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.696409] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.698285] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.703073] iommu: Default domain type: Passthrough [ 0.705398] SCSI subsystem initialized [ 0.706125] ACPI: bus type USB registered [ 0.707113] usbcore: registered new interface driver usbfs [ 0.709081] usbcore: registered new interface driver hub [ 0.710073] usbcore: registered new device driver usb [ 0.712133] pps_core: LinuxPPS API ver. 1 registered [ 0.713011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.716112] PTP clock support registered [ 0.718170] EDAC MC: Ver: 3.0.0 [ 0.719764] PCI: Using ACPI for IRQ routing [ 0.720962] NetLabel: Initializing [ 0.721015] NetLabel: domain hash size = 128 [ 0.722012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.723243] NetLabel: unlabeled traffic allowed by default [ 0.725147] vgaarb: loaded [ 0.727373] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.729022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.738000] clocksource: Switched to clocksource kvm-clock [ 0.849927] VFS: Disk quotas dquot_6.6.0 [ 0.851620] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.857045] *** VALIDATE ramfs *** [ 0.857854] *** VALIDATE hugetlbfs *** [ 0.859661] pnp: PnP ACPI init [ 0.861662] pnp: PnP ACPI: found 6 devices [ 0.893580] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.896920] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.899671] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.902454] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.905286] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.908221] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.911666] NET: Registered protocol family 2 [ 0.914726] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.919684] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.924159] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.930275] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.933552] TCP: Hash tables configured (established 65536 bind 65536) [ 0.938903] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.941605] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.944059] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.946718] NET: Registered protocol family 1 [ 0.949432] RPC: Registered named UNIX socket transport module. [ 0.951743] RPC: Registered udp transport module. [ 0.954099] RPC: Registered tcp transport module. [ 0.955844] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.959435] NET: Registered protocol family 44 [ 0.961429] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.964256] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.967089] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.969690] PCI: CLS 0 bytes, default 64 [ 0.972610] Unpacking initramfs... [ 2.436982] debug: unmapping init [mem 0xffff8b8abcc54000-0xffff8b8abffbffff] [ 2.442391] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.445272] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.450242] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.931621] Initialise system trusted keyrings [ 2.932986] Key type blacklist registered [ 2.934693] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.945352] zbud: loaded [ 2.948961] *** VALIDATE nfs *** [ 2.950153] *** VALIDATE nfs4 *** [ 2.951621] pstore: using deflate compression [ 2.955332] Platform Keyring initialized [ 3.068380] NET: Registered protocol family 38 [ 3.070577] Key type asymmetric registered [ 3.072442] Asymmetric key parser 'x509' registered [ 3.074599] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.077938] io scheduler mq-deadline registered [ 3.082078] io scheduler kyber registered [ 3.084181] io scheduler bfq registered [ 3.086908] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.090399] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.093793] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.097373] ACPI: Power Button [PWRF] [ 3.103672] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.111217] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.125161] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.131578] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.184347] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.212726] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.242976] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.248399] Non-volatile memory driver v1.3 [ 3.250541] Linux agpgart interface v0.103 [ 3.284865] virtio_blk virtio1: [vda] 139280 512-byte logical blocks (71.3 MB/68.0 MiB) [ 3.289111] vda: detected capacity change from 0 to 71311360 [ 3.305096] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.308547] vdb: detected capacity change from 0 to 1073741824 [ 3.325132] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.328451] vdc: detected capacity change from 0 to 2621440000 [ 3.344151] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.346674] vdd: detected capacity change from 0 to 2621440000 [ 3.362370] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.365107] vde: detected capacity change from 0 to 4294967296 [ 3.382100] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.384823] vdf: detected capacity change from 0 to 4294967296 [ 3.396303] libphy: Fixed MDIO Bus: probed [ 3.403784] usbcore: registered new interface driver usbserial_generic [ 3.406741] usbserial: USB Serial support registered for generic [ 3.409559] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.415171] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.417264] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.420328] mousedev: PS/2 mouse device common for all mice [ 3.424551] rtc_cmos 00:05: RTC can wake from S4 [ 3.429632] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.434175] rtc_cmos 00:05: registered as rtc0 [ 3.435983] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.438389] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.441751] intel_pstate: CPU model not supported [ 3.444058] hid: raw HID events driver (C) Jiri Kosina [ 3.451867] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.452489] usbcore: registered new interface driver usbhid [ 3.458152] usbhid: USB HID core driver [ 3.459333] drop_monitor: Initializing network drop monitor service [ 3.460730] Initializing XFRM netlink socket [ 3.462035] NET: Registered protocol family 10 [ 3.464576] Segment Routing with IPv6 [ 3.465761] NET: Registered protocol family 17 [ 3.467208] mpls_gso: MPLS GSO support [ 3.472642] RAS: Correctable Errors collector initialized. [ 3.475376] AVX version of gcm_enc/dec engaged. [ 3.477502] AES CTR mode by8 optimization enabled [ 3.556861] sched_clock: Marking stable (3556780951, 0)->(4499981958, -943201007) [ 3.560196] registered taskstats version 1 [ 3.561924] Loading compiled-in X.509 certificates [ 3.564102] zswap: loaded using pool lzo/zbud [ 3.590521] Key type big_key registered [ 3.605682] Key type encrypted registered [ 3.607825] ima: No TPM chip found, activating TPM-bypass! [ 3.610219] ima: Allocated hash algorithm: sha1 [ 3.612167] ima: No architecture policies found [ 3.614235] evm: Initialising EVM extended attributes: [ 3.616405] evm: security.selinux [ 3.617857] evm: security.ima [ 3.619237] evm: security.capability [ 3.620711] evm: HMAC attrs: 0x1 [ 3.623343] rtc_cmos 00:05: setting system clock to 2026-06-16 10:16:25 UTC (1781604985) [ 3.630988] debug: unmapping init [mem 0xffffffffac603000-0xffffffffac7fffff] [ 3.634848] debug: unmapping init [mem 0xffffffffab382000-0xffffffffab658fff] [ 3.644081] Write protecting the kernel read-only data: 28672k [ 3.647784] debug: unmapping init [mem 0xffffffffa9a03000-0xffffffffa9bfffff] [ 3.650994] debug: unmapping init [mem 0xffffffffaa314000-0xffffffffaa3fffff] [ 3.686946] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.695926] systemd[1]: Detected virtualization kvm. [ 3.697652] systemd[1]: Detected architecture x86-64. [ 3.699613] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.721556] systemd[1]: No hostname configured. [ 3.723359] systemd[1]: Set hostname to . [ 3.724984] random: systemd: uninitialized urandom read (16 bytes read) [ 3.727419] systemd[1]: Initializing machine ID from random generator. [ 3.771805] random: ln: uninitialized urandom read (6 bytes read) [ 3.855535] random: systemd: uninitialized urandom read (16 bytes read) [ 3.860581] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.867781] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.872436] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.552166] device-mapper: uevent: version 1.0.3 [ 4.554497] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel [ 5.182062] random: fast init done Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.420593] virtio_net virtio0 ens2: renamed from eth0 [ 5.431093] scsi host0: ata_piix [ 5.447180] hrtimer: interrupt took 4004112 ns [ 5.503169] scsi host1: ata_piix [ 5.504983] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.507731] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.380926] dracut-initqueue[584]: RTNETLINK answers: File exists [ 10.126469] random: crng init done [ 10.128432] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.772385] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.064119] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.360725] SELinux: Disabled at runtime. [ 12.423384] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.429838] systemd[1]: Detected virtualization kvm. [ 12.431294] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.986410] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.990056] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.995115] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.999995] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.003607] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.013886] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.035518] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 13.095387] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt.[ 13.574203] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.913313] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.967037] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.066353] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.076762] EDAC sbridge: Ver: 1.1.2 [ 15.895456] Key type dns_resolver registered [ 16.198793] NFS: Registering the id_resolver key type [ 16.200974] Key type id_resolver registered [ 16.202679] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg406-server login: [ 42.934558] libcfs: loading out-of-tree module taints kernel. [ 42.963293] Key type ._llcrypt registered [ 42.964942] Key type .llcrypt registered [ 43.019255] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 57.466568] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 59.374749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 59.386470] alg: No test for adler32 (adler32-zlib) [ 60.756939] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 61.981491] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 63.879217] Key type lgssc registered [ 66.151899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 86.422160] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 128.399967] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 142.145110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 142.182250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 143.505773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 143.571709] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 143.694740] Lustre: lustre-MDT0000: new disk, initializing [ 143.854786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 143.886483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 148.495995] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 161.698592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 161.825681] Lustre: 6512:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 161.881057] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 161.886762] Lustre: Skipped 1 previous similar message [ 162.017585] Lustre: lustre-MDT0001: new disk, initializing [ 162.120899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 162.158106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 162.167108] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 166.348453] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.763626] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 180.058973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 180.357266] Lustre: lustre-OST0000: new disk, initializing [ 180.360915] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 180.368632] Lustre: 8452:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 180.506295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 182.263018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 182.278278] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 182.377831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 187.078733] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.196431] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 202.319535] Lustre: lustre-OST0001: new disk, initializing [ 202.323243] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 202.329294] Lustre: 9526:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 202.400720] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 209.098416] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.000375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 212.010456] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 212.067059] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 221.619535] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 228.172301] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 235.515332] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing check_logdir /tmp/testlogs/ [ 240.841945] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing yml_node [ 245.363947] Lustre: DEBUG MARKER: Client: 2.17.54.40 [ 247.755565] Lustre: DEBUG MARKER: MDS: 2.17.54.40 [ 249.753615] Lustre: DEBUG MARKER: OSS: 2.17.54.40 [ 251.197176] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 16 06:20:30 EDT 2026 [ 267.590426] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 269.588605] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 288.737400] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.740582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 288.764290] Lustre: Skipped 1 previous similar message [ 288.784973] Lustre: Skipped 3 previous similar messages [ 292.909669] Lustre: server umount lustre-MDT0000 complete [ 296.727239] LustreError: 9525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781605278 with bad export cookie 9968664412305468079 [ 296.735131] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 296.745177] LustreError: 9525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 298.978460] LustreError: 8460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.979268] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.987400] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 299.020400] LustreError: 8460:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 299.059021] Lustre: Skipped 3 previous similar messages [ 301.985233] LustreError: 12109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 301.988871] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 302.005481] LustreError: 12109:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 302.018285] Lustre: Skipped 2 previous similar messages [ 307.109805] LustreError: 12108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.110151] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 307.132175] LustreError: 12108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 307.155262] Lustre: Skipped 1 previous similar message [ 311.263272] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 311.516505] Lustre: server umount lustre-MDT0001 complete [ 315.201297] Lustre: server umount lustre-OST0000 complete [ 318.785830] Lustre: server umount lustre-OST0001 complete [ 325.648110] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 335.926313] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 370.654627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 370.932643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 370.976293] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 371.085746] Lustre: lustre-MDT0000: new disk, initializing [ 371.218565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 371.269264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 375.890669] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 386.566550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 386.706617] Lustre: 15085:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 386.759743] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 386.763275] Lustre: Skipped 1 previous similar message [ 386.826911] Lustre: lustre-MDT0001: new disk, initializing [ 386.935610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 386.998737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 387.006218] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 391.406369] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 400.173910] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 404.878382] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 410.936561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 411.325151] Lustre: lustre-OST0000: new disk, initializing [ 411.332876] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 411.339692] Lustre: 17053:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 411.485728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 412.892546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 412.913934] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 412.991296] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 417.588694] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 426.532058] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 430.964870] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 431.215904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 435.943176] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 436.302962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 438.756697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 438.758154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 438.770581] LustreError: Skipped 1 previous similar message [ 438.772474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 438.795293] Lustre: Skipped 1 previous similar message [ 438.831700] Lustre: Skipped 1 previous similar message [ 444.693688] Lustre: server umount lustre-OST0000 complete [ 448.482296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 448.491440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 448.504672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 448.510557] Lustre: Skipped 2 previous similar messages [ 454.736890] Lustre: server umount lustre-MDT0000 complete [ 458.477043] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781605440 with bad export cookie 9968664412305469612 [ 458.482265] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 458.489377] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 459.018763] Lustre: server umount lustre-MDT0001 complete [ 463.168988] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 06:24:02 (1781605442) [ 465.377145] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 466.919666] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 06:24:06 (1781605446) [ 472.678225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.017279] LustreError: 19464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 473.093472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.888559] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 478.186710] LustreError: 19465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 484.639550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.201630] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 496.475736] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 501.452661] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 508.113760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 514.620791] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 522.051437] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 528.363101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 528.368940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.373732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 528.390330] Lustre: Skipped 2 previous similar messages [ 528.401458] Lustre: Skipped 4 previous similar messages [ 534.374183] Lustre: server umount lustre-OST0000 complete [ 538.593057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 538.610508] Lustre: Skipped 1 previous similar message [ 546.260150] Lustre: server umount lustre-MDT0000 complete [ 548.836794] LustreError: 19465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.852671] LustreError: 19465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 549.693482] LustreError: 19445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781605531 with bad export cookie 9968664412305470494 [ 549.695085] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 549.704372] LustreError: 19445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 550.050936] Lustre: server umount lustre-MDT0001 complete [ 560.379503] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 563.449200] Key type lgssc unregistered [ 563.912577] LNet: 22994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 563.927559] LNetError: 22994:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 563.957250] LNet: Removed LNI 192.168.204.106@tcp [ 565.121822] Key type .llcrypt unregistered [ 565.128555] Key type ._llcrypt unregistered [ 588.693605] Key type ._llcrypt registered [ 588.696943] Key type .llcrypt registered [ 588.825329] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 589.847115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 589.863700] alg: No test for adler32 (adler32-zlib) [ 590.982823] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 591.262966] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 592.911566] Key type lgssc registered [ 594.027611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 602.302705] loop: module loaded [ 602.395660] loop0: detected capacity change from 0 to 204800000 [ 602.765900] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 602.776578] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 603.146258] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 609.695714] loop0: detected capacity change from 0 to 204800000 [ 609.997166] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 610.028402] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 610.046248] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 615.595740] loop0: detected capacity change from 0 to 204800000 [ 621.810338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 621.844097] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.393969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 623.556525] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 623.786832] Lustre: lustre-MDT0000: new disk, initializing [ 623.952831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 623.998601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 628.039195] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 636.737393] loop1: detected capacity change from 0 to 204800000 [ 642.791395] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 643.089791] Lustre: lustre-OST07c6: new disk, initializing [ 643.092180] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 643.097269] Lustre: Skipped 1 previous similar message [ 643.101396] Lustre: 26176:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 643.189992] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 650.246918] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 653.416832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 653.437362] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 653.478974] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 655.528951] Lustre: 25164:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 655.544463] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 655.553914] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 655.561519] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 655.577700] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 655.582918] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 672.981253] Lustre: 25164:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 673.000257] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 673.006769] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 673.018506] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 673.026818] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 673.032519] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 677.720707] Lustre: 25162:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 677.732420] Lustre: 25162:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 677.740619] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 677.748253] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 677.752776] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 677.757023] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 677.765973] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 677.773189] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 677.786377] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 677.789267] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 677.794082] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 677.797468] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 680.952123] Lustre: 25163:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 680.961402] Lustre: 25163:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 680.968242] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 680.979276] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 680.987689] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 681.000197] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 681.014745] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 681.022425] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 681.031232] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 681.040130] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 681.049291] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 681.055220] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.719223] Lustre: 26783:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 685.733542] Lustre: 26783:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 685.742373] Lustre: 26783:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 685.758162] Lustre: 26783:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.770223] Lustre: 26783:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 685.777732] Lustre: 26783:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.783875] Lustre: 26783:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 685.791311] Lustre: 26783:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.795899] Lustre: 26783:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 685.800887] Lustre: 26783:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.807945] Lustre: 26783:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 685.818825] Lustre: 26783:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.944478] Lustre: 25162:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 696.953952] Lustre: 25162:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 696.959934] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 696.967172] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 696.974070] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 696.979520] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 696.984367] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 696.989651] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 696.994352] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 697.001103] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 697.009098] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 697.015416] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 745.441926] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 745.463532] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 750.561485] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 755.680894] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 758.239292] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 758.358039] Lustre: server umount lustre-OST07c6 complete [ 768.741674] Lustre: server umount lustre-MDT0000 complete [ 785.351426] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 788.391889] Key type lgssc unregistered [ 788.674084] LNet: 28738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.682322] LNetError: 28738:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 789.733644] LNet: Removed LNI 192.168.204.106@tcp [ 790.709183] Key type .llcrypt unregistered [ 790.713818] Key type ._llcrypt unregistered [ 810.123988] Key type ._llcrypt registered [ 810.127669] Key type .llcrypt registered [ 810.220237] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 811.542460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 811.584854] alg: No test for adler32 (adler32-zlib) [ 812.596958] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 812.835560] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 814.531681] Key type lgssc registered [ 815.734818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 823.953147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 823.968788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 825.468664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 830.249772] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 836.074019] LustreError: 30008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 836.092595] LustreError: 30008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 839.335107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 839.814262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 844.786492] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 852.457526] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 857.969677] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 865.792288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 866.225737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 870.361225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 873.135299] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 881.970958] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 901.089431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 901.097464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 901.119224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 905.193254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 905.201345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 905.203809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 907.229964] Lustre: server umount lustre-OST0000 complete [ 915.427066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 915.451658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 915.459177] Lustre: Skipped 1 previous similar message [ 918.767455] Lustre: server umount lustre-MDT0000 complete [ 920.547654] LustreError: 30008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 920.558976] LustreError: 30008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 922.428339] LustreError: 29988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781605904 with bad export cookie 12802441180001332005 [ 922.429997] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 922.439490] LustreError: 29988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 923.028828] Lustre: server umount lustre-MDT0001 complete [ 934.870174] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 938.439192] Key type lgssc unregistered [ 938.817049] LNet: 33744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.823975] LNetError: 33744:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 938.855061] LNet: Removed LNI 192.168.204.106@tcp [ 939.690927] Key type .llcrypt unregistered [ 939.693589] Key type ._llcrypt unregistered [ 960.050549] Key type ._llcrypt registered [ 960.054238] Key type .llcrypt registered [ 960.149352] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 961.548358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 961.590045] alg: No test for adler32 (adler32-zlib) [ 962.789165] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 963.095710] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 964.881365] Key type lgssc registered [ 966.035680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 973.920679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 973.941305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 975.456680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 980.450514] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 985.568980] LustreError: 35018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 985.590348] LustreError: 35018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 989.205794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 989.604980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 993.515712] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1000.621588] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1006.859377] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1013.820143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1014.212292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1020.420830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1020.853486] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1028.281208] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1039.842349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1039.856291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1039.869213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1040.869303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1040.874334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1040.876175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1040.879462] Lustre: Skipped 1 previous similar message [ 1045.105979] Lustre: server umount lustre-OST0000 complete [ 1051.106032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1051.107415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1051.134441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1051.141529] Lustre: Skipped 1 previous similar message [ 1056.225997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1056.238757] Lustre: Skipped 1 previous similar message [ 1061.346320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1061.352201] Lustre: Skipped 1 previous similar message [ 1063.391365] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1063.590078] Lustre: server umount lustre-MDT0000 complete [ 1065.447225] LustreError: 35017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1065.465547] LustreError: 35017:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1067.168846] LustreError: 34997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781606049 with bad export cookie 13542816803431734426 [ 1067.180116] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1067.686521] Lustre: server umount lustre-MDT0001 complete [ 1079.514814] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1082.863811] Key type lgssc unregistered [ 1083.238737] LNet: 38551:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.249552] LNetError: 38551:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1083.289724] LNet: Removed LNI 192.168.204.106@tcp [ 1084.314631] Key type .llcrypt unregistered [ 1084.322484] Key type ._llcrypt unregistered [ 1110.439802] Key type ._llcrypt registered [ 1110.446497] Key type .llcrypt registered [ 1110.589547] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1111.912948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1111.934292] alg: No test for adler32 (adler32-zlib) [ 1112.983510] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 1113.310972] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1115.060100] Key type lgssc registered [ 1116.288836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1125.609553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1125.623109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1127.054525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1132.270780] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1136.614672] LustreError: 39972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1136.635560] LustreError: 39972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1141.731142] LustreError: 39971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1141.852603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1142.239894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1146.578683] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1153.867537] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1159.239794] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1167.349792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1167.876875] LustreError: 41792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1168.312798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1171.379661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1174.569562] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1182.002748] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1188.322366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1188.339972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1188.366920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1189.535838] Lustre: server umount lustre-MDT0000 complete [ 1191.917497] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.925044] LustreError: 39972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1191.956927] LustreError: 39972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1193.168261] LustreError: 39952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781606175 with bad export cookie 2530503790106558756 [ 1193.175413] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1193.183413] LustreError: 39952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1193.896955] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.6@tcp (stopping) [ 1193.901373] Lustre: Skipped 1 previous similar message [ 1197.025055] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1197.025884] LustreError: 39966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1197.037587] Lustre: Skipped 1 previous similar message [ 1197.041172] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1197.051492] LustreError: 39966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1199.728153] Lustre: server umount lustre-MDT0001 complete [ 1220.816459] Lustre: server umount lustre-OST0000 complete [ 1237.970510] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1241.005652] Key type lgssc unregistered [ 1241.349106] LNet: 43830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1241.365427] LNetError: 43830:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1241.390785] LNet: Removed LNI 192.168.204.106@tcp [ 1242.363189] Key type .llcrypt unregistered [ 1242.366166] Key type ._llcrypt unregistered [ 1268.174966] Key type ._llcrypt registered [ 1268.181636] Key type .llcrypt registered [ 1268.285220] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1269.486487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1269.756246] alg: No test for adler32 (adler32-zlib) [ 1270.881665] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 1271.141398] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1272.895600] Key type lgssc registered [ 1274.309843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1283.093521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1283.108376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1284.704709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1289.515798] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1294.825248] LustreError: 45249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1294.862581] LustreError: 45249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1298.756509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1299.221383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1303.865674] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1311.207913] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1317.016530] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1323.764238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1323.953076] LustreError: 47074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1324.177406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1329.351874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1331.102279] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1339.171316] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1344.994561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1344.994872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1345.011199] LustreError: Skipped 1 previous similar message [ 1345.025036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1345.964533] Lustre: lustre-OST0000: Not available for connect from 192.168.204.6@tcp (stopping) [ 1345.978679] Lustre: Skipped 1 previous similar message [ 1350.116784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1350.124445] Lustre: Skipped 1 previous similar message [ 1355.236796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1355.251189] Lustre: Skipped 2 previous similar messages [ 1359.327131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1359.508257] Lustre: server umount lustre-OST0000 complete [ 1380.838294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1380.854474] Lustre: Skipped 1 previous similar message [ 1380.858708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1380.868861] Lustre: Skipped 1 previous similar message [ 1385.616811] Lustre: server umount lustre-MDT0000 complete [ 1385.953102] LustreError: 46682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1385.969781] LustreError: 46682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1389.457638] LustreError: 45232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781606371 with bad export cookie 15463896050879142877 [ 1389.461346] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1389.467150] LustreError: 45232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1395.927125] Lustre: server umount lustre-MDT0001 complete [ 1406.685265] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1409.678445] Key type lgssc unregistered [ 1410.106888] LNet: 48965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1410.119940] LNetError: 48965:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1410.143801] LNet: Removed LNI 192.168.204.106@tcp [ 1411.042887] Key type .llcrypt unregistered [ 1411.044706] Key type ._llcrypt unregistered [ 1433.763488] Key type ._llcrypt registered [ 1433.766125] Key type .llcrypt registered [ 1433.851754] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1434.590121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1434.650737] alg: No test for adler32 (adler32-zlib) [ 1435.711328] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 1435.904331] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1437.639241] Key type lgssc registered [ 1439.047745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1447.019744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1447.042879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1448.541140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1453.238501] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1458.669591] LustreError: 50379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1458.682244] LustreError: 50379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1461.687389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1462.166561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1467.231068] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1475.118688] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1480.621794] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1487.640301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1487.991838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1493.890694] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1494.154531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1501.481202] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1510.841638] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1514.582216] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1526.894532] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-16 6h41m45s [ 1528.399189] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1532.392057] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1532.401425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1532.405865] Lustre: Skipped 2 previous similar messages [ 1532.425841] Lustre: Skipped 2 previous similar messages [ 1533.475775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1533.488870] Lustre: Skipped 2 previous similar messages [ 1534.878744] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.6@tcp (stopping) [ 1536.974948] Lustre: server umount lustre-MDT0000 complete [ 1539.558206] LustreError: 50379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1539.584790] LustreError: 50379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1540.590406] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1542.016038] LustreError: 51551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1545.698102] LustreError: 51370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1545.714905] LustreError: 51370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1547.385147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1547.497698] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1547.756534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.810855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1548.783483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1552.096339] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1555.599565] Lustre: DEBUG MARKER: Wait for df (36510) ... [ 1557.383042] Lustre: DEBUG MARKER: done [ 1563.111694] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1563.114678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1563.126107] LustreError: Skipped 1 previous similar message [ 1563.131695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1563.144928] Lustre: Skipped 1 previous similar message [ 1563.172326] Lustre: Skipped 2 previous similar messages [ 1567.858400] Lustre: server umount lustre-OST0000 complete [ 1574.367995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1574.372658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1574.377216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1574.377224] Lustre: Skipped 2 previous similar messages [ 1574.406016] Lustre: Skipped 1 previous similar message [ 1577.859509] Lustre: server umount lustre-MDT0000 complete [ 1581.537843] LustreError: 51370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1581.564057] LustreError: 51370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1581.593150] LustreError: 50359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781606563 with bad export cookie 3105088703921873087 [ 1581.615283] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1581.965543] Lustre: server umount lustre-MDT0001 complete [ 1592.545425] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1595.165914] Key type lgssc unregistered [ 1595.452587] LNet: 55306:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1595.466863] LNetError: 55306:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1596.524183] LNet: Removed LNI 192.168.204.106@tcp [ 1597.434370] Key type .llcrypt unregistered [ 1597.436051] Key type ._llcrypt unregistered [ 1636.760436] Key type ._llcrypt registered [ 1636.764860] Key type .llcrypt registered [ 1636.902238] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1638.224576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1638.256775] alg: No test for adler32 (adler32-zlib) [ 1639.436043] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 1639.792771] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1641.519220] Key type lgssc registered [ 1642.798386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1650.279843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1650.302209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1651.574334] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1651.599788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1651.809218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1657.088353] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1665.978732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1666.152379] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1666.178638] Lustre: 57517:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1666.433136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1671.378663] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1678.701498] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1683.866293] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1690.887905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1691.114838] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1691.400955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1697.633844] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1698.925333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1703.936959] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1710.392871] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-16 6h44m50s [ 1711.612110] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1717.406669] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1719.675619] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1719.726092] LustreError: 59386:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1740.251766] Lustre: lustre-MDT0000: Client 3f05c280-8456-4b59-8289-2713ab6360c8 (at 192.168.204.6@tcp) reconnecting [ 1759.736730] LustreError: 59386:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1761.842689] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1764.460799] Lustre: DEBUG MARKER: done [ 1770.469440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1770.486553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1770.514165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1770.979229] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1775.864551] Lustre: server umount lustre-OST0000 complete [ 1781.217071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1781.223975] Lustre: Skipped 1 previous similar message [ 1781.227477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1781.233381] Lustre: Skipped 2 previous similar messages [ 1782.240794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1785.885262] Lustre: server umount lustre-MDT0000 complete [ 1789.191196] LustreError: 57507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781606771 with bad export cookie 5683126419910145368 [ 1789.196244] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1789.210764] LustreError: 57507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1789.539757] Lustre: server umount lustre-MDT0001 complete [ 1799.818754] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1802.418510] Key type lgssc unregistered [ 1802.748130] LNet: 61844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1802.758928] LNetError: 61844:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1802.776510] LNet: Removed LNI 192.168.204.106@tcp [ 1803.604204] Key type .llcrypt unregistered [ 1803.606126] Key type ._llcrypt unregistered [ 1844.190149] Key type ._llcrypt registered [ 1844.193364] Key type .llcrypt registered [ 1844.327304] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1845.326803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1845.535607] alg: No test for adler32 (adler32-zlib) [ 1846.571543] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 1846.837102] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1848.567173] Key type lgssc registered [ 1850.067476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1854.049591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1854.068868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1855.313671] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1855.364802] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1855.573687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1856.527509] Lustre: Failing over lustre-MDT0000 [ 1856.818989] Lustre: server umount lustre-MDT0000 complete [ 1863.043289] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 06:47:22 (1781606842) [ 1869.216806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1869.686440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1874.554468] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1882.433119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1882.577828] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1882.623741] Lustre: 64641:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1882.839302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1887.158940] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1893.726478] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1898.386861] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1904.405555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1904.584937] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1904.730448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1910.449551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1910.653553] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1917.290569] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1923.075497] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1929.188370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1929.197762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1929.206638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1930.438685] Lustre: server umount lustre-MDT0000 complete [ 1931.243189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1931.260887] Lustre: Skipped 1 previous similar message [ 1933.468356] LustreError: 64635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781606915 with bad export cookie 4126410186400711186 [ 1933.477086] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1933.484230] LustreError: 64635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1933.773703] Lustre: server umount lustre-MDT0001 complete [ 1936.683459] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1940.144557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1946.817315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1952.735432] Lustre: 63568:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781606918/real 1781606918] req@00000000bc5b6daa x1868150161314688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781606934 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1952.767983] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1954.783305] Lustre: 63567:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781606920/real 1781606920] req@0000000078650723 x1868150161314816/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781606936 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1957.855490] Lustre: 63565:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781606923/real 1781606923] req@000000006cbf41e3 x1868150161314944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781606939 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1958.947156] LustreError: 63564:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000041361f66 x1868150161315840/t0(0) o250->MGC192.168.204.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1959.168514] LustreError: 66492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1959.186970] LustreError: 66492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1959.297245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1959.438252] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1960.351274] Lustre: 63565:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781606925/real 1781606925] req@00000000f4e67a8c x1868150161315200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781606941 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1962.351904] LustreError: 66493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1962.372795] LustreError: 66493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1963.851919] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1964.383101] Lustre: 63567:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781606930/real 1781606930] req@000000009dcfb872 x1868150161315328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781606946 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1967.589786] LustreError: 68408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1967.606652] LustreError: 68408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1971.642934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.692601] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1971.694534] LustreError: 68409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1971.722198] LustreError: 68409:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1972.006670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1976.288222] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1977.321090] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1982.661721] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1987.486249] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1995.217104] Lustre: server umount lustre-MDT0000 complete [ 1997.793889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1997.808129] LustreError: 66493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1998.277257] LustreError: 64635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781606980 with bad export cookie 4126410186400713461 [ 1998.282031] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1998.288160] LustreError: 64635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1998.563803] Lustre: server umount lustre-MDT0001 complete [ 2003.925991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2011.485284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2018.272672] Lustre: 63568:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781606984/real 1781606984] req@00000000ed59874d x1868150161340288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781607000 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2018.291871] Lustre: 63568:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2018.297726] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.303853] Lustre: Skipped 2 previous similar messages [ 2023.459974] LustreError: 63564:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b97b3a05 x1868150161341440/t0(0) o250->MGC192.168.204.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2023.716176] LustreError: 66492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2023.726305] LustreError: 66492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2023.763644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2023.810885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2027.302786] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2034.020967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2034.148424] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2034.156463] Lustre: Skipped 1 previous similar message [ 2037.364855] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2037.420377] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2042.905469] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2047.204851] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2053.093237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2053.094672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2053.105868] Lustre: Skipped 2 previous similar messages [ 2053.110121] Lustre: Skipped 1 previous similar message [ 2054.639847] Lustre: server umount lustre-MDT0000 complete [ 2057.337532] LustreError: 65388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781607039 with bad export cookie 4126410186400714840 [ 2057.339147] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2057.346119] LustreError: 65388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2057.620313] Lustre: server umount lustre-MDT0001 complete [ 2062.762199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2065.359767] Lustre: DEBUG MARKER: files compared the same [ 2077.252384] Lustre: server umount lustre-OST0000 complete [ 2089.979916] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2092.405759] Key type lgssc unregistered [ 2092.691702] LNet: 74366:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2092.698219] LNetError: 74366:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2092.713777] LNet: Removed LNI 192.168.204.106@tcp [ 2093.411473] Key type .llcrypt unregistered [ 2093.414884] Key type ._llcrypt unregistered [ 2111.280280] Key type ._llcrypt registered [ 2111.281743] Key type .llcrypt registered [ 2111.358761] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2111.945973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2111.975125] alg: No test for adler32 (adler32-zlib) [ 2112.974244] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 2113.191240] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2114.855193] Key type lgssc registered [ 2115.856640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2122.101369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2122.123091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2123.563922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2127.074757] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug +malloc all [ 2133.421164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.621594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2136.771221] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug +malloc all [ 2142.101417] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2146.163890] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2150.865878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2151.064483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2155.457492] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug +malloc all [ 2160.422928] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2169.516639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2179.050417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2179.057060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2179.075687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2180.065708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2180.067646] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2180.077873] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.933556] Lustre: server umount lustre-OST0000 complete [ 2189.792430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2189.802477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.825085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2189.832534] Lustre: Skipped 1 previous similar message [ 2192.051612] Lustre: server umount lustre-MDT0000 complete [ 2194.882380] LustreError: 75745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781607176 with bad export cookie 12588885073073346945 [ 2194.886862] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2194.900781] LustreError: 75745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2195.140549] Lustre: server umount lustre-MDT0001 complete [ 2203.178643] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2205.293399] Key type lgssc unregistered [ 2205.520224] LNet: 79313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2205.528437] LNetError: 79313:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2205.549502] LNet: Removed LNI 192.168.204.106@tcp [ 2206.201024] Key type .llcrypt unregistered [ 2206.206709] Key type ._llcrypt unregistered [ 2222.720748] Key type ._llcrypt registered [ 2222.724174] Key type .llcrypt registered [ 2222.801180] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2223.411044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2223.455462] alg: No test for adler32 (adler32-zlib) [ 2224.437876] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 2224.607975] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2226.248454] Key type lgssc registered [ 2227.105264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2232.819386] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2232.853542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2248.415995] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2253.535345] LustreError: 80680:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.106@tcp: failed processing log, type 4: rc = -110 [ 2280.223225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2284.483371] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2289.254528] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2294.973933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2295.091351] Lustre: *** cfs_fail_loc=706, val=0*** [ 2295.362200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2295.418190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2298.535276] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2300.386227] LustreError: 80705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2300.406482] LustreError: 80705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2304.611312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2304.859928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2307.580120] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2312.206477] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2316.052372] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2319.334852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2319.336931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2319.347934] Lustre: Skipped 1 previous similar message [ 2319.354132] Lustre: Skipped 1 previous similar message [ 2323.725933] Lustre: server umount lustre-OST0000 complete [ 2329.571554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2329.575740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2329.585613] Lustre: Skipped 1 previous similar message [ 2329.595049] Lustre: Skipped 1 previous similar message [ 2332.359297] Lustre: server umount lustre-MDT0000 complete [ 2334.578088] LustreError: 80686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781607316 with bad export cookie 8908653775311551884 [ 2334.582542] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2334.586018] LustreError: 80686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2334.769421] Lustre: server umount lustre-MDT0001 complete [ 2341.571371] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2343.464453] Key type lgssc unregistered [ 2343.676716] LNet: 84290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2343.682983] LNetError: 84290:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2344.742371] LNet: Removed LNI 192.168.204.106@tcp [ 2345.363158] Key type .llcrypt unregistered [ 2345.367810] Key type ._llcrypt unregistered [ 2361.436995] Key type ._llcrypt registered [ 2361.439314] Key type .llcrypt registered [ 2361.519345] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2361.988768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2362.084838] alg: No test for adler32 (adler32-zlib) [ 2363.070522] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 2363.269259] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2364.903154] Key type lgssc registered [ 2365.694988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2370.748168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2370.763397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.636244] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2379.349442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2379.680573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2382.267518] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2384.864996] LustreError: 86372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2384.878484] LustreError: 86372:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2387.709077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2387.900885] LustreError: 86375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2387.918899] LustreError: 86375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2387.930925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.986144] LustreError: 86376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2390.001382] LustreError: 86376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2391.845538] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2395.105176] LustreError: 86375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.115700] LustreError: 86375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2397.126328] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2397.136911] Lustre: MGS: Client 1981aff1-c7f6-499d-988e-8b870932134e (at 0@lo) reconnecting [ 2397.151171] LustreError: 86386:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2397.297789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2397.343621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2400.142778] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2412.516587] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2412.525351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2412.536811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2413.032289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2413.046939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2413.054713] Lustre: Skipped 1 previous similar message [ 2416.503156] Lustre: server umount lustre-OST0000 complete [ 2423.266798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2423.266851] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2423.273940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2423.290263] Lustre: Skipped 1 previous similar message [ 2425.563784] Lustre: server umount lustre-MDT0000 complete [ 2428.050889] Lustre: server umount lustre-MDT0001 complete [ 2430.498494] Lustre: server umount MGS complete [ 2437.715943] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 06:56:57 (1781607417) [ 2454.052996] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 2459.221122] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2479.586779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2482.095970] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2488.026051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2488.176677] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2488.207948] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2488.260320] Lustre: lustre-MDT0001: new disk, initializing [ 2488.306038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2488.320664] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2488.331199] Lustre: 93533:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2488.339267] LustreError: 93533:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2490.694331] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2496.980097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2497.148177] Lustre: lustre-OST0000: new disk, initializing [ 2497.152500] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2497.158188] Lustre: 94298:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2497.204585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2501.159286] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2506.785587] Lustre: 94960:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000842b066b x1868150703428480/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781607504 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2506.864865] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2510.745767] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2510.755060] Lustre: MGS: Client a111840f-0ebb-4022-90f4-7708ed4583bc (at 0@lo) reconnecting [ 2510.783934] Lustre: 92800:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2510.794856] LustreError: 93518:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2510.811149] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2510.856716] Lustre: lustre-MDT0000: new disk, initializing [ 2510.941769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2510.957518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2512.045648] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2512.048316] Lustre: 95365:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000611b8d36 x1868150703434240/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1781607509 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2512.074453] Lustre: 95365:0:(client.c:1638:after_reply()) Skipped 1 previous similar message [ 2513.442288] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2514.085050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2514.090761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2514.102227] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2534.369138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2534.372081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2534.381793] Lustre: Skipped 1 previous similar message [ 2534.387327] Lustre: Skipped 2 previous similar messages [ 2539.488878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2539.495075] Lustre: Skipped 1 previous similar message [ 2540.147868] Lustre: server umount lustre-OST0000 complete [ 2544.608321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2544.608494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2544.620239] Lustre: Skipped 1 previous similar message [ 2548.327841] Lustre: server umount lustre-MDT0000 complete [ 2549.730866] LustreError: 93508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.742215] LustreError: 93508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2556.712814] Lustre: server umount lustre-MDT0001 complete [ 2558.848981] Lustre: server umount MGS complete [ 2564.980547] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 06:59:05 (1781607545) [ 2574.465597] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2576.099963] Key type lgssc unregistered [ 2576.278432] LNet: 97860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2576.286424] LNetError: 97860:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2576.300275] LNet: Removed LNI 192.168.204.106@tcp [ 2576.790707] Key type .llcrypt unregistered [ 2576.792678] Key type ._llcrypt unregistered [ 2587.337515] Key type ._llcrypt registered [ 2587.339837] Key type .llcrypt registered [ 2587.406601] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2588.103802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2588.118084] alg: No test for adler32 (adler32-zlib) [ 2589.070918] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 2589.221057] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2590.847458] Key type lgssc registered [ 2591.587675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2596.566787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2596.590813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2597.910430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2600.611685] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2606.010885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2606.257243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2608.750877] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2612.868772] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2616.090221] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2620.187192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2620.370315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2622.453420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 2623.762913] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2627.988297] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2637.793368] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2637.798887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.809525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2641.890467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2641.892329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2641.901468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2647.013585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2647.023899] Lustre: Skipped 2 previous similar messages [ 2652.127256] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2652.130711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2652.143190] Lustre: Skipped 1 previous similar message [ 2652.218333] Lustre: server umount lustre-OST0000 complete [ 2657.249193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2657.251313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2657.257382] Lustre: Skipped 1 previous similar message [ 2660.395568] Lustre: server umount lustre-MDT0000 complete [ 2662.371253] LustreError: 99061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2662.394657] LustreError: 99061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2662.715383] LustreError: 100901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781607644 with bad export cookie 16679252045456803025 [ 2662.733118] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2662.939875] Lustre: server umount lustre-MDT0001 complete [ 2668.695834] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2670.283196] Key type lgssc unregistered [ 2670.460120] LNet: 102591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2670.463727] LNetError: 102591:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2670.472383] LNet: Removed LNI 192.168.204.106@tcp [ 2670.954366] Key type .llcrypt unregistered [ 2670.956527] Key type ._llcrypt unregistered [ 2680.206240] Key type ._llcrypt registered [ 2680.208745] Key type .llcrypt registered [ 2680.282973] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2680.987501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2681.013137] alg: No test for adler32 (adler32-zlib) [ 2681.973563] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 2682.118283] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2683.751438] Key type lgssc registered [ 2684.466085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2688.077890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2688.086311] LustreError: 103664:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 2688.090614] LustreError: 103667:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 2688.095971] LustreError: 103664:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 2688.099862] LustreError: 103664:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2688.104877] LustreError: 103664:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2688.110429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2689.417203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2693.681791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2693.878348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2696.137630] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2699.374938] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing lsmod [ 2700.974970] LustreError: 104871:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 2700.979508] LustreError: 104872:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 2700.985225] LustreError: 104871:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 2700.988291] LustreError: 104871:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2700.997151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2700.997575] LustreError: 104871:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2701.219533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2703.842281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2703.847305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2703.849289] Lustre: Skipped 2 previous similar messages [ 2703.853224] Lustre: Skipped 1 previous similar message [ 2708.964414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2708.971655] Lustre: Skipped 2 previous similar messages [ 2714.082287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2714.085350] Lustre: Skipped 2 previous similar messages [ 2717.151466] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2717.219260] Lustre: server umount lustre-MDT0000 complete [ 2719.201724] LustreError: 103704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2719.217284] LustreError: 103704:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2719.236689] LustreError: 103684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781607701 with bad export cookie 538202793757010802 [ 2719.244589] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2719.248570] LustreError: 103684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2720.224861] LustreError: 104135:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000de123274 x1868151036857984/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2720.337689] Lustre: server umount lustre-MDT0001 complete [ 2732.603299] Lustre: server umount lustre-OST0000 complete [ 2737.097547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2737.341561] LustreError: 105936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2737.381897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2739.548350] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2742.754116] LustreError: 105937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2743.934984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2744.139695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2746.340362] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2750.216293] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2753.006669] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2756.643255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2756.840748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2760.007813] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2762.981343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 2763.850451] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2773.472214] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2773.476155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2773.484147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2774.496689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2774.500734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2778.348638] Lustre: server umount lustre-OST0000 complete [ 2784.737226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2784.738208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2784.744501] Lustre: Skipped 1 previous similar message [ 2784.746849] Lustre: Skipped 2 previous similar messages [ 2786.687323] Lustre: server umount lustre-MDT0000 complete [ 2788.522523] LustreError: 105963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781607770 with bad export cookie 538202793757011481 [ 2788.525606] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2788.532232] LustreError: 105963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2788.689992] Lustre: server umount lustre-MDT0001 complete [ 2793.804996] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2795.268934] Key type lgssc unregistered [ 2795.451752] LNet: 109468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2795.455807] LNetError: 109468:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2795.466471] LNet: Removed LNI 192.168.204.106@tcp [ 2795.865677] Key type .llcrypt unregistered [ 2795.867951] Key type ._llcrypt unregistered [ 2808.458957] Key type ._llcrypt registered [ 2808.460093] Key type .llcrypt registered [ 2808.526595] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2808.926079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2808.935480] alg: No test for adler32 (adler32-zlib) [ 2809.837113] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 2809.952270] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2811.551182] Key type lgssc registered [ 2812.153349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2816.235881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2816.249138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2817.497530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2819.644933] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2824.235728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2824.455912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2826.646245] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2830.374652] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2833.105786] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2836.523243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2836.688680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2839.642221] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2842.861820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 2843.343966] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2855.776613] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2857.475526] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2860.515661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2860.517804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2860.518197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2860.522435] Lustre: Skipped 1 previous similar message [ 2865.073585] Lustre: server umount lustre-OST0000 complete [ 2870.753516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2870.754618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2870.761902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2870.773858] Lustre: Skipped 2 previous similar messages [ 2873.236523] Lustre: server umount lustre-MDT0000 complete [ 2875.369292] LustreError: 110842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781607857 with bad export cookie 16417349622381400266 [ 2875.375338] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2875.377469] LustreError: 110842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2875.611663] Lustre: server umount lustre-MDT0001 complete [ 2881.543259] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2883.018929] Key type lgssc unregistered [ 2883.179945] LNet: 114515:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2883.184955] LNetError: 114515:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2883.196731] LNet: Removed LNI 192.168.204.106@tcp [ 2883.634821] Key type .llcrypt unregistered [ 2883.636580] Key type ._llcrypt unregistered [ 2892.585624] Key type ._llcrypt registered [ 2892.586977] Key type .llcrypt registered [ 2892.649944] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2893.017942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2893.048063] alg: No test for adler32 (adler32-zlib) [ 2893.955587] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 2894.066669] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2895.663165] Key type lgssc registered [ 2896.221608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2900.384824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2900.396502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2901.680682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2903.646985] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2907.989675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2908.223438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2910.276683] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2914.185466] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2916.932623] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2920.451119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2920.626827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2923.650608] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2926.761806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 2927.920137] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2937.313219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2937.323062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2937.338277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2939.362443] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2939.362505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2939.368202] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2943.186360] Lustre: server umount lustre-OST0000 complete [ 2949.088274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2949.093505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2949.104060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2949.110611] Lustre: Skipped 1 previous similar message [ 2951.286367] Lustre: server umount lustre-MDT0000 complete [ 2953.340711] LustreError: 115697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781607935 with bad export cookie 5243050509915274351 [ 2953.342876] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2953.354845] LustreError: 115697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2953.524237] Lustre: server umount lustre-MDT0001 complete [ 2958.670427] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2960.107428] Key type lgssc unregistered [ 2960.265853] LNet: 119243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2960.274274] LNetError: 119243:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2960.291829] LNet: Removed LNI 192.168.204.106@tcp [ 2960.706588] Key type .llcrypt unregistered [ 2960.709963] Key type ._llcrypt unregistered [ 2973.836600] Key type ._llcrypt registered [ 2973.838384] Key type .llcrypt registered [ 2973.887869] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2974.337394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2974.399292] alg: No test for adler32 (adler32-zlib) [ 2975.322523] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 2975.453718] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2977.071154] Key type lgssc registered [ 2977.778645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2982.317108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2982.333242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.632187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2986.032676] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2990.943598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2991.202801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2993.644604] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2997.910588] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3000.915848] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3004.683143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3004.899613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3008.334492] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3009.000604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3012.372624] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3021.659273] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3023.716779] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3023.722153] Lustre: Skipped 2 previous similar messages [ 3026.314200] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3026.318219] Lustre: Skipped 2 previous similar messages [ 3028.000737] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3028.015683] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3030.966742] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.6@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3030.971060] Lustre: Skipped 2 previous similar messages [ 3037.189838] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.6@tcp 192.168.0.[2,10]@tcp [ 3040.345360] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.6/24@tcp in log lustre-MDT0000 [ 3040.353968] Lustre: Skipped 2 previous similar messages [ 3043.093597] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.6/24@tcp [ 3043.097053] Lustre: Skipped 1 previous similar message [ 3046.881627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3046.886506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3046.893301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3046.898561] Lustre: Skipped 1 previous similar message [ 3052.021591] Lustre: server umount lustre-OST0000 complete [ 3057.121931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3057.123697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3057.128490] Lustre: Skipped 2 previous similar messages [ 3057.138533] Lustre: Skipped 1 previous similar message [ 3062.241579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3062.246167] Lustre: Skipped 1 previous similar message [ 3066.339108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3066.343619] Lustre: Skipped 1 previous similar message [ 3068.384162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3068.459985] Lustre: server umount lustre-MDT0000 complete [ 3070.105383] LustreError: 123264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781608051 with bad export cookie 11394138171984033961 [ 3070.108149] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3070.111127] LustreError: 123264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3070.258411] Lustre: server umount lustre-MDT0001 complete [ 3074.666708] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3075.897178] Key type lgssc unregistered [ 3076.047776] LNet: 125566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3076.051445] LNetError: 125566:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3076.057433] LNet: Removed LNI 192.168.204.106@tcp [ 3076.431057] Key type .llcrypt unregistered [ 3076.432688] Key type ._llcrypt unregistered [ 3089.547631] Key type ._llcrypt registered [ 3089.549057] Key type .llcrypt registered [ 3089.596671] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3090.004525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3090.041650] alg: No test for adler32 (adler32-zlib) [ 3090.951279] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 3091.064246] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3092.671293] Key type lgssc registered [ 3093.180068] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3096.742579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3096.754632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3097.956462] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3097.967601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3097.969705] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.6/24@tcp [ 3099.885153] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3103.901311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3104.062979] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3104.074090] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.6/24@tcp [ 3104.099664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3105.923799] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3109.513754] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3112.119418] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3115.480541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3115.664161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3118.201866] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3121.255965] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3122.855878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 3138.528363] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3138.531099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3138.532445] LustreError: Skipped 1 previous similar message [ 3138.539182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3139.552787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3139.555771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3139.558086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3139.565640] Lustre: Skipped 1 previous similar message [ 3142.263351] Lustre: server umount lustre-OST0000 complete [ 3144.675629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3144.676801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3144.682651] Lustre: Skipped 1 previous similar message [ 3144.690777] Lustre: Skipped 1 previous similar message [ 3149.792182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3149.794909] Lustre: Skipped 1 previous similar message [ 3150.197661] Lustre: server umount lustre-MDT0000 complete [ 3151.800803] LustreError: 127086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781608133 with bad export cookie 3794387697354562361 [ 3151.803615] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3151.810519] LustreError: 127086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3151.943649] Lustre: server umount lustre-MDT0001 complete [ 3156.431839] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3157.796502] Key type lgssc unregistered [ 3157.963798] LNet: 130708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3157.967617] LNetError: 130708:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3157.976604] LNet: Removed LNI 192.168.204.106@tcp [ 3158.373923] Key type .llcrypt unregistered [ 3158.376049] Key type ._llcrypt unregistered [ 3169.872141] Key type ._llcrypt registered [ 3169.873391] Key type .llcrypt registered [ 3169.929086] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 3177.908421] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3178.551887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3178.571906] alg: No test for adler32 (adler32-zlib) [ 3179.489979] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 3179.598976] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3181.215304] Key type lgssc registered [ 3181.730704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3197.972639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3197.990673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3199.130164] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3199.149996] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3199.201512] Lustre: lustre-MDT0000: new disk, initializing [ 3199.256651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3199.274371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3201.243863] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3206.473648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3206.528343] Lustre: 134087:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3206.558816] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3206.562162] Lustre: Skipped 1 previous similar message [ 3206.603405] Lustre: lustre-MDT0001: new disk, initializing [ 3206.656219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3206.673168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3206.678593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3208.520287] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3213.407960] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3216.026651] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3218.931965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3219.058501] Lustre: lustre-OST0000: new disk, initializing [ 3219.062590] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3219.066598] Lustre: 136055:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3219.116978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3221.669875] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3224.068321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3224.072559] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3224.110063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3226.384633] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3228.252681] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3228.350234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3230.327871] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3230.429336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3234.275572] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3234.276114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3234.285261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3234.288080] Lustre: Skipped 1 previous similar message [ 3237.432988] Lustre: server umount lustre-OST0000 complete [ 3239.395242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3239.396583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3239.407495] Lustre: Skipped 2 previous similar messages [ 3244.512520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3244.515572] Lustre: Skipped 2 previous similar messages [ 3245.248504] Lustre: server umount lustre-MDT0000 complete [ 3246.806037] LustreError: 136054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781608228 with bad export cookie 5187776539356071854 [ 3246.807549] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3246.810652] LustreError: 136054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3246.962201] Lustre: server umount lustre-MDT0001 complete [ 3250.497092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3250.743042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3252.498278] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3255.776203] LustreError: 138267:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3255.782184] LustreError: 138267:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3256.040497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3256.192729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3257.821585] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3260.963655] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3263.175073] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3265.978810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3266.135343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3268.589673] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3271.609976] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3273.487227] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3275.633281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3277.306847] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3277.400419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3282.536249] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3282.624687] Lustre: lustre-OST0001: new disk, initializing [ 3282.627059] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3282.630744] Lustre: 141652:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3282.671571] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3285.320054] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3289.107566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3289.112631] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3289.147839] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3289.230352] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3289.323700] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3291.038428] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3291.133221] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3293.445061] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a1d051d6000.ost_server_uuid 50 [ 3294.123463] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9a1d051d6000.ost_server_uuid in FULL state after 0 sec [ 3299.300679] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3299.300817] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3299.303983] LustreError: Skipped 2 previous similar messages [ 3299.311131] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3299.315736] Lustre: Skipped 1 previous similar message [ 3302.983209] Lustre: server umount lustre-OST0001 complete [ 3304.419690] LustreError: 141001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3304.431436] LustreError: 141001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3306.464405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3306.471782] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3306.478575] Lustre: Skipped 1 previous similar message [ 3306.482959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3308.512512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3309.539338] LustreError: 140087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3309.545024] LustreError: 140087:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3310.805777] Lustre: server umount lustre-OST0000 complete [ 3312.608638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3312.613812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3312.620145] Lustre: Skipped 1 previous similar message [ 3314.657356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3314.664349] Lustre: Skipped 5 previous similar messages [ 3318.587155] Lustre: server umount lustre-MDT0000 complete [ 3319.776452] LustreError: 138262:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3319.787509] LustreError: 138262:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3320.122390] LustreError: 138249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781608301 with bad export cookie 5187776539356072743 [ 3320.126635] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3320.127343] LustreError: 138249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3320.300468] Lustre: server umount lustre-MDT0001 complete [ 3328.374105] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3329.692897] Key type lgssc unregistered [ 3329.855278] LNet: 144649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3329.860123] LNetError: 144649:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3329.871631] LNet: Removed LNI 192.168.204.106@tcp [ 3330.229409] Key type .llcrypt unregistered [ 3330.231209] Key type ._llcrypt unregistered [ 3346.748392] Key type ._llcrypt registered [ 3346.751783] Key type .llcrypt registered [ 3346.812132] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 3353.831517] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3354.347961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3354.357615] alg: No test for adler32 (adler32-zlib) [ 3355.229701] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 3355.331764] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3356.935196] Key type lgssc registered [ 3357.429173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3371.816203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3371.823906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3372.936402] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3372.954116] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3372.991993] Lustre: lustre-MDT0000: new disk, initializing [ 3373.027813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3373.037774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3374.681417] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3379.670027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3379.719840] Lustre: 148774:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3379.745915] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3379.748880] Lustre: Skipped 1 previous similar message [ 3379.787891] Lustre: lustre-MDT0001: new disk, initializing [ 3379.830678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3379.845570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3379.849682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3381.376491] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3385.618246] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3387.653995] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3390.472938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3390.629473] Lustre: lustre-OST0000: new disk, initializing [ 3390.632606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3390.638388] Lustre: 150743:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3390.682986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3392.992235] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3394.617260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3394.621643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3394.666740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3397.274699] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3400.581912] Lustre: Failing over lustre-OST0000 [ 3400.648704] Lustre: server umount lustre-OST0000 complete [ 3414.186550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3414.343340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3414.350652] Lustre: Skipped 1 previous similar message [ 3414.355073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3414.369483] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3414.552380] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3416.764872] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3418.365928] Lustre: Failing over lustre-MDT0000 [ 3418.478411] Lustre: server umount lustre-MDT0000 complete [ 3419.617199] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3419.626196] Lustre: Skipped 2 previous similar messages [ 3420.639245] Lustre: 147003:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781608386/real 1781608386] req@0000000041361f66 x1868151743605504/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781608402 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3424.736259] LustreError: 148785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3424.746988] LustreError: 148785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3425.761342] Lustre: 147005:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781608391/real 1781608391] req@00000000c5bf26bc x1868151743607424/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781608407 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3429.857774] LustreError: 148786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3429.867515] LustreError: 148786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3432.269680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3432.329834] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3432.341319] Lustre: 148777:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.106@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3432.434459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3432.459059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3432.481417] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3434.142574] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3435.389324] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3437.543921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3437.554239] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3437.584922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3442.659066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3442.659445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3442.662918] LustreError: Skipped 1 previous similar message [ 3442.669488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3442.673760] Lustre: Skipped 1 previous similar message [ 3444.951563] Lustre: server umount lustre-OST0000 complete [ 3447.778572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3447.780614] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3447.783152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3447.789345] Lustre: Skipped 2 previous similar messages [ 3452.526166] Lustre: server umount lustre-MDT0000 complete [ 3452.897791] LustreError: 148786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3452.906790] LustreError: 148786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3454.101522] LustreError: 148765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781608435 with bad export cookie 1521476915123148390 [ 3454.103794] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3454.106877] LustreError: 148765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3454.239614] Lustre: server umount lustre-MDT0001 complete [ 3458.543888] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3459.778701] Key type lgssc unregistered [ 3459.924644] LNet: 154393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3459.928836] LNetError: 154393:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3459.938412] LNet: Removed LNI 192.168.204.106@tcp [ 3460.324095] Key type .llcrypt unregistered [ 3460.325972] Key type ._llcrypt unregistered [ 3482.342218] Key type ._llcrypt registered [ 3482.343711] Key type .llcrypt registered [ 3482.399712] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3482.805813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3482.848097] alg: No test for adler32 (adler32-zlib) [ 3483.743062] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 3483.842414] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3485.439280] Key type lgssc registered [ 3485.905983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3489.482279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3489.490591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3490.750270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3492.470614] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3496.755337] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3498.974382] LustreError: 155941:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3498.982537] LustreError: 155941:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3498.987647] LustreError: 155941:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 3499.130074] Lustre: server umount lustre-MDT0000 complete [ 3505.353243] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 3509.267834] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3524.044950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3524.158813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3524.176217] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3524.216959] Lustre: lustre-MDT0000: new disk, initializing [ 3524.255813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3524.265474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3525.805771] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3530.818964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3530.865375] Lustre: 159239:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3530.894335] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3530.897231] Lustre: Skipped 1 previous similar message [ 3530.935148] Lustre: lustre-MDT0001: new disk, initializing [ 3530.985366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3531.003155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3531.009576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3532.888400] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3537.128103] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3539.132894] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3541.667327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3541.779980] Lustre: lustre-OST0000: new disk, initializing [ 3541.782297] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3541.786609] Lustre: 161205:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3541.823068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3543.941540] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3548.295544] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3549.189606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3549.194107] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3549.234591] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3551.317598] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3552.080048] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 07:15:32 (1781608532) [ 3651.551904] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3651.552121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3651.562628] Lustre: Skipped 2 previous similar messages [ 3656.671825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3656.675132] Lustre: Skipped 4 previous similar messages [ 3661.791761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3661.794362] Lustre: Skipped 2 previous similar messages [ 3663.327176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3663.401280] Lustre: server umount lustre-MDT0000 complete [ 3664.881612] LustreError: 159232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781608646 with bad export cookie 9898435525409079627 [ 3664.882791] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3664.887083] LustreError: 159232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3666.912236] LustreError: 159246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3666.913642] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3666.927129] LustreError: 159246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3666.934891] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3669.919703] LustreError: 159247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3675.039804] LustreError: 160587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3675.039822] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3675.051975] Lustre: Skipped 1 previous similar message [ 3679.199144] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3679.291322] Lustre: server umount lustre-MDT0001 complete [ 3687.059317] Lustre: server umount lustre-OST0000 complete [ 3690.474293] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 3694.563930] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3708.481679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3708.588852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3708.605745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3708.646351] Lustre: lustre-MDT0000: new disk, initializing [ 3708.683816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3708.694730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3710.229374] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3714.881205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3714.931213] Lustre: 165838:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3714.956351] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3714.959190] Lustre: Skipped 1 previous similar message [ 3714.999387] Lustre: lustre-MDT0001: new disk, initializing [ 3715.051416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3715.056307] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3716.572174] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3720.650151] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3722.750762] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3725.367775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3725.490867] Lustre: lustre-OST0000: new disk, initializing [ 3725.493525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3725.496465] Lustre: 167807:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3725.531957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3725.535016] Lustre: Skipped 1 previous similar message [ 3727.792599] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3731.455968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3731.460568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3731.499142] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3731.916688] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3733.611106] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3733.711155] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3735.588541] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3735.685413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3736.544982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3736.547477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3736.549708] LustreError: Skipped 1 previous similar message [ 3736.553515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3736.557841] Lustre: Skipped 1 previous similar message [ 3738.607184] Lustre: server umount lustre-OST0000 complete [ 3740.642068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3740.646758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3746.461677] Lustre: server umount lustre-MDT0000 complete [ 3746.785682] LustreError: 166774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3746.794363] LustreError: 166774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3748.078274] LustreError: 165831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781608729 with bad export cookie 9898435525409238457 [ 3748.081202] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3748.083811] LustreError: 165831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3748.227055] Lustre: server umount lustre-MDT0001 complete [ 3751.775350] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:18:52 (1781608732) [ 3765.937657] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 3777.175499] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3817.079637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3817.354298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3817.406898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3817.514300] Lustre: lustre-MDT0000: new disk, initializing [ 3817.616208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3817.645812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3821.510579] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3830.038443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3830.186813] Lustre: 172563:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3830.404605] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3834.521934] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3842.750654] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3847.124415] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3852.590744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3852.842396] Lustre: lustre-OST0000: new disk, initializing [ 3852.845209] Lustre: Skipped 1 previous similar message [ 3852.848524] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3852.852395] Lustre: Skipped 2 previous similar messages [ 3852.856433] Lustre: 174542:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3852.945529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3852.950776] Lustre: Skipped 1 previous similar message [ 3857.356661] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3862.094557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3862.111436] Lustre: Skipped 1 previous similar message [ 3862.124954] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3862.261639] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3866.849178] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3877.856317] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3877.856806] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3877.860129] LustreError: Skipped 1 previous similar message [ 3877.867366] Lustre: Skipped 1 previous similar message [ 3877.870712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3877.875989] Lustre: Skipped 5 previous similar messages [ 3880.716120] Lustre: server umount lustre-OST0000 complete [ 3886.560050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3886.562520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3898.342771] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3898.487501] Lustre: server umount lustre-MDT0000 complete [ 3901.595128] LustreError: 172554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781608883 with bad export cookie 9898435525409239325 [ 3901.611647] LustreError: 172554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3901.623584] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3901.860076] Lustre: server umount lustre-MDT0001 complete [ 3908.086689] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:21:27 (1781608887) [ 3922.656476] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 3932.115688] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3964.816973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3965.083641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3965.116292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3965.191405] Lustre: lustre-MDT0000: new disk, initializing [ 3965.303266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3965.331410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3968.557914] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3976.921564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3977.082197] Lustre: 178862:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3977.450880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3981.089210] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3988.304595] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3993.245963] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3998.854065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3999.119785] Lustre: 180840:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4003.600515] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4005.004383] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4011.031314] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4018.531992] Lustre: server umount lustre-OST0000 complete [ 4023.776455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4023.788076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4023.805876] Lustre: Skipped 1 previous similar message [ 4023.809838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4023.816899] Lustre: Skipped 7 previous similar messages [ 4032.044754] Lustre: server umount lustre-MDT0000 complete [ 4034.967048] LustreError: 178853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609016 with bad export cookie 9898435525409240410 [ 4034.974406] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4034.976366] LustreError: 178853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4044.451840] Lustre: server umount lustre-MDT0001 complete [ 4053.805096] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 4056.165615] Key type lgssc unregistered [ 4056.464148] LNet: 182695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4056.471991] LNetError: 182695:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4056.481817] LNet: Removed LNI 192.168.204.106@tcp [ 4057.150134] Key type .llcrypt unregistered [ 4057.157661] Key type ._llcrypt unregistered [ 4075.238701] Key type ._llcrypt registered [ 4075.241675] Key type .llcrypt registered [ 4075.341224] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4075.897416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4075.932873] alg: No test for adler32 (adler32-zlib) [ 4076.959544] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 4077.168821] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 4078.847209] Key type lgssc registered [ 4079.752639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4086.069915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4086.083733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4087.538235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4091.424967] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4098.016800] LustreError: 184089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4098.035604] LustreError: 184089:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4098.549795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4098.782921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4102.450595] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4107.802339] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4111.938752] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4117.637489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4117.945638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4122.824796] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4128.611677] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4132.270945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4152.806501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4152.809205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4152.817576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4152.822908] Lustre: Skipped 1 previous similar message [ 4156.152966] Lustre: server umount lustre-OST0000 complete [ 4160.484757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4160.493097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4160.518378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4160.522673] Lustre: Skipped 1 previous similar message [ 4163.041351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4163.044463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4168.162565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4168.167038] Lustre: Skipped 2 previous similar messages [ 4172.263194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4172.270239] Lustre: Skipped 1 previous similar message [ 4173.279140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4173.420465] Lustre: server umount lustre-MDT0000 complete [ 4176.334487] LustreError: 184069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609158 with bad export cookie 1946346137270398764 [ 4176.346950] LustreError: 184069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4176.347119] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4176.651694] Lustre: server umount lustre-MDT0001 complete [ 4185.975414] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 4188.180593] Key type lgssc unregistered [ 4188.452817] LNet: 187645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4188.456444] LNetError: 187645:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4188.468607] LNet: Removed LNI 192.168.204.106@tcp [ 4189.118418] Key type .llcrypt unregistered [ 4189.124779] Key type ._llcrypt unregistered [ 4207.940454] Key type ._llcrypt registered [ 4207.942422] Key type .llcrypt registered [ 4208.032215] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4208.731623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4208.877048] alg: No test for adler32 (adler32-zlib) [ 4209.903601] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 4210.092550] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 4211.727170] Key type lgssc registered [ 4212.807642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4219.126650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4219.152927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4220.608177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4224.284404] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4230.665560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4231.000337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4234.281939] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4239.772787] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4243.664893] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4249.559469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4249.790626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4255.019708] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4261.068674] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4266.163874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4271.588562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4271.590346] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4271.594476] LustreError: Skipped 1 previous similar message [ 4271.605053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4271.611565] Lustre: Skipped 1 previous similar message [ 4273.049935] Lustre: lustre-OST0000: Not available for connect from 192.168.204.6@tcp (stopping) [ 4276.709714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4276.717046] Lustre: Skipped 1 previous similar message [ 4277.611427] Lustre: server umount lustre-OST0000 complete [ 4283.375889] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9a1d18090800.ost_server_uuid 50 [ 4284.579833] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4297.699356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4297.717330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4297.733248] Lustre: Skipped 1 previous similar message [ 4297.738582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4301.856220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4301.861172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4303.637728] Lustre: server umount lustre-MDT0000 complete [ 4306.635277] LustreError: 189020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609288 with bad export cookie 1425312784616875 [ 4306.636626] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4306.644018] LustreError: 189020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4306.962782] Lustre: server umount lustre-MDT0001 complete [ 4313.519347] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 07:28:13 (1781609293) [ 4318.919518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4319.344283] LustreError: 192951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4319.362738] LustreError: 192951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4319.402689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4323.192258] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4324.832366] LustreError: 192952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4329.368586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4329.647751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4333.124076] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4338.349624] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4342.261933] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4347.148053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4347.397319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4351.720522] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4356.725786] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4361.431799] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4361.600185] Lustre: lustre-OST0001: new disk, initializing [ 4361.603215] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4361.607652] Lustre: 195800:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4365.578468] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4365.811419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4368.959046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4368.967226] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4369.061207] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4371.725841] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4374.500843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4374.512098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4374.516844] Lustre: Skipped 2 previous similar messages [ 4379.545958] Lustre: server umount lustre-OST0000 complete [ 4379.625208] LustreError: 194773:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4379.638515] LustreError: 194773:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4382.603994] LustreError: 196339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4383.331230] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4383.490732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4386.309249] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4386.448982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4387.713234] LustreError: 194772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4387.724496] LustreError: 194772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4400.096757] LustreError: 194772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4400.098406] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4400.103896] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4400.103905] Lustre: Skipped 1 previous similar message [ 4400.104558] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4400.104562] Lustre: Skipped 1 previous similar message [ 4400.113163] LustreError: 194772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4400.141372] LustreError: Skipped 1 previous similar message [ 4404.565423] Lustre: server umount lustre-OST0001 complete [ 4410.347959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4410.356314] Lustre: Skipped 2 previous similar messages [ 4413.238940] Lustre: server umount lustre-MDT0000 complete [ 4415.840502] LustreError: 195799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609397 with bad export cookie 1425312784617946 [ 4415.847915] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4415.856903] LustreError: 195799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4416.103541] Lustre: server umount lustre-MDT0001 complete [ 4430.256175] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 07:30:10 (1781609410) [ 4433.569176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4433.733020] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4433.757157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4433.889186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4433.893347] Lustre: Skipped 1 previous similar message [ 4435.874362] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4440.430651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4440.490982] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4440.515372] Lustre: 199236:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4442.889267] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4446.440128] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4449.107719] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4452.805721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4452.933560] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4454.218871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4456.547352] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4460.920513] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4464.859849] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4464.900358] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4468.668892] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4473.233913] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4477.413652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4477.418466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4477.428370] Lustre: Skipped 1 previous similar message [ 4477.435653] Lustre: Skipped 4 previous similar messages [ 4481.098511] Lustre: server umount lustre-OST0000 complete [ 4482.530497] LustreError: 201086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4482.546172] LustreError: 201086:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4500.404548] Lustre: server umount lustre-OST0001 complete [ 4517.343828] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4517.439155] Lustre: server umount lustre-MDT0000 complete [ 4519.744535] LustreError: 199227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609501 with bad export cookie 1425312784619346 [ 4519.748293] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4519.759277] LustreError: 199227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4519.975420] Lustre: server umount lustre-MDT0001 complete [ 4535.263846] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 07:31:55 (1781609515) [ 4544.192505] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 4549.989333] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4570.066282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4570.207249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4570.227882] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4570.294667] Lustre: lustre-MDT0000: new disk, initializing [ 4570.372427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4570.379423] Lustre: Skipped 3 previous similar messages [ 4570.392707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4572.891057] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4578.925918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4578.989697] Lustre: 207327:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4579.030738] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4579.034032] Lustre: Skipped 1 previous similar message [ 4579.080784] Lustre: lustre-MDT0001: new disk, initializing [ 4579.153920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4579.164834] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4581.761810] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4587.894728] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4591.537628] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4595.680412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4595.872452] Lustre: lustre-OST0000: new disk, initializing [ 4595.875765] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4595.882335] Lustre: 209292:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4596.380684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4596.390214] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4596.476605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4600.088618] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4607.184276] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4610.938862] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4611.120736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4614.461919] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4614.667941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4617.186605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4617.193285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4617.196513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4617.202675] Lustre: Skipped 5 previous similar messages [ 4617.209410] Lustre: Skipped 12 previous similar messages [ 4622.314523] LustreError: 209337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4622.339278] LustreError: 209337:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 4622.451555] Lustre: server umount lustre-OST0000 complete [ 4625.376322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4625.380175] LustreError: Skipped 1 previous similar message [ 4633.486579] Lustre: server umount lustre-MDT0000 complete [ 4636.088066] LustreError: 207318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609617 with bad export cookie 1425312784620907 [ 4636.092602] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4636.098984] LustreError: 207318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4636.300259] Lustre: server umount lustre-MDT0001 complete [ 4642.653335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4643.077581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4643.085722] Lustre: Skipped 2 previous similar messages [ 4646.353996] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4652.596369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4656.262658] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4661.146694] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4664.803553] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4669.579848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4673.700504] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4678.627599] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4681.939501] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4688.416675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 4691.653429] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4691.792987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4693.474493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4693.485459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4693.494292] Lustre: Skipped 2 previous similar messages [ 4699.337322] Lustre: server umount lustre-OST0000 complete [ 4703.369624] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4705.944374] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4731.813779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4735.566858] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4737.187650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4737.205276] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4739.419945] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4747.430604] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4747.439080] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4752.913775] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 12 sec [ 4757.988470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4757.989566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4757.992295] LustreError: Skipped 2 previous similar messages [ 4757.995593] Lustre: Skipped 6 previous similar messages [ 4761.078794] Lustre: server umount lustre-OST0000 complete [ 4769.155657] Lustre: server umount lustre-MDT0000 complete [ 4771.200495] LustreError: 211491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609753 with bad export cookie 1425312784621789 [ 4771.203117] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4771.205386] LustreError: 211491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4771.386084] Lustre: server umount lustre-MDT0001 complete [ 4775.623754] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 07:35:55 (1781609755) [ 4779.191966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4779.448967] LustreError: 217532:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4779.460608] LustreError: 217532:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4779.494988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4779.498539] Lustre: Skipped 3 previous similar messages [ 4781.803080] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4786.359430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4788.756602] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4792.992646] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4795.987535] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4799.482827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4802.983207] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4807.234581] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4809.596896] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4809.746571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4812.060186] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4812.168142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4815.639114] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4815.723330] Lustre: lustre-OST0001: new disk, initializing [ 4815.726139] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4815.731029] Lustre: 220898:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4818.414126] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4822.554698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4822.561024] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4822.600541] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4823.281153] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4825.210042] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4825.325783] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4827.231083] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4827.332831] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4832.737133] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4832.743625] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4832.752281] Lustre: Skipped 6 previous similar messages [ 4834.323351] Lustre: server umount lustre-OST0001 complete [ 4837.212569] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4837.317763] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4839.445099] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4839.551171] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4865.714289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4870.687735] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4871.153371] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4871.172784] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4871.178160] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4871.193980] Lustre: Skipped 1 previous similar message [ 4879.231906] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4880.591369] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4888.309323] Lustre: server umount lustre-OST0001 complete [ 4904.418229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4904.429165] LustreError: Skipped 3 previous similar messages [ 4909.827293] Lustre: server umount lustre-OST0000 complete [ 4918.546671] Lustre: server umount lustre-MDT0000 complete [ 4921.493725] LustreError: 220897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609903 with bad export cookie 1425312784622755 [ 4921.494790] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4921.500875] LustreError: 220897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4927.813715] Lustre: server umount lustre-MDT0001 complete [ 4941.589247] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 07:38:41 (1781609921) [ 4945.471890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4945.647842] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4945.671235] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4948.520599] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4954.179978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4954.264079] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4954.285287] Lustre: 226135:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4957.022625] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4961.148627] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4964.242333] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4968.348520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4968.469822] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4973.074187] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4978.115179] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4983.943607] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4984.005720] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4984.060454] LustreError: 227978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4984.067740] LustreError: 227978:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 28 previous similar messages [ 4987.527422] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4992.072916] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4993.168526] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4996.062796] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4996.206463] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4998.859650] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4998.978473] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5002.568494] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a1d05008800.ost_server_uuid 50 [ 5003.637839] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9a1d05008800.ost_server_uuid in FULL state after 0 sec [ 5004.325833] Lustre: Permanently deactivating lustre-OST0001 [ 5004.336509] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5006.553640] Lustre: Permanently reactivating lustre-OST0001 [ 5011.220483] Lustre: server umount lustre-OST0001 complete [ 5017.057421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5017.065780] Lustre: Skipped 18 previous similar messages [ 5028.319334] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5028.413478] Lustre: server umount lustre-OST0000 complete [ 5045.215487] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5045.364615] Lustre: server umount lustre-MDT0000 complete [ 5048.034879] LustreError: 226127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781610029 with bad export cookie 1425312784624043 [ 5048.035916] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5048.058291] LustreError: 226127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5048.391857] Lustre: server umount lustre-MDT0001 complete [ 5064.210116] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 07:40:44 (1781610044) [ 5068.799125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5069.005953] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5069.034783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5069.040480] Lustre: Skipped 5 previous similar messages [ 5069.171437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5069.176183] Lustre: Skipped 8 previous similar messages [ 5071.583494] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5076.647574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5076.719506] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5076.749358] Lustre: 232823:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5079.565602] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5083.917966] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5087.478711] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5092.039160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5092.173730] Lustre: Permanently deactivating lustre-OST0000 [ 5092.187281] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5096.168248] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5100.037396] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5100.542106] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5105.070572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5108.404763] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5109.265098] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5113.930147] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5114.938426] Lustre: Permanently reactivating lustre-OST0000 [ 5114.952806] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5114.958479] Lustre: Skipped 2 previous similar messages [ 5124.635978] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5124.646467] Lustre: Skipped 12 previous similar messages [ 5124.650593] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5124.655906] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5124.666617] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5124.696515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5124.723514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5129.185713] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5129.194828] LustreError: Skipped 6 previous similar messages [ 5133.965313] Lustre: server umount lustre-OST0001 complete [ 5150.175127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5150.309577] Lustre: server umount lustre-OST0000 complete [ 5159.152054] Lustre: server umount lustre-MDT0000 complete [ 5161.668688] LustreError: 232815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781610143 with bad export cookie 1425312784625688 [ 5161.671764] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5161.679192] LustreError: 232815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5161.934526] Lustre: server umount lustre-MDT0001 complete [ 5168.712910] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5170.421817] Key type lgssc unregistered [ 5170.593940] LNet: 237701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5170.597923] LNetError: 237701:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5170.609881] LNet: Removed LNI 192.168.204.106@tcp [ 5171.090181] Key type .llcrypt unregistered [ 5171.092184] Key type ._llcrypt unregistered [ 5185.779324] Key type ._llcrypt registered [ 5185.781823] Key type .llcrypt registered [ 5185.863973] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5186.588461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5186.600459] alg: No test for adler32 (adler32-zlib) [ 5187.566796] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 5187.685618] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 5189.295159] Key type lgssc registered [ 5189.867575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5194.978868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5194.999378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5196.364391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5199.046574] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5204.533302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5204.606326] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5204.627392] Lustre: 239132:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5204.634866] Lustre: Permanently deactivating lustre-MDT0001 [ 5204.653193] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5204.886643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5207.382511] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5211.542211] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5213.681441] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5214.230881] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5217.797830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5217.986175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5221.443980] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5222.118316] LustreError: 240984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5222.124865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5222.126830] LustreError: 240984:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5224.987317] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5227.492869] LustreError: 240986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5228.428779] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5228.545319] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5229.548199] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 5229.548457] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5229.559960] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5231.527750] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5235.453586] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5237.519356] Lustre: Permanently reactivating lustre-MDT0001 [ 5237.530935] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5237.539291] Lustre: Skipped 2 previous similar messages [ 5237.656122] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5237.671138] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 5237.684288] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 5237.691343] LustreError: 242706:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 5237.693301] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5237.697577] LustreError: 242706:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 5237.707595] LustreError: 242706:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.106@tcp: cfg command failed: rc = -108 [ 5237.713038] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 5237.713038] [ 5237.717714] LustreError: 239136:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 5244.301823] Lustre: lustre-MDT0001: Client ac511856-e721-482e-847c-767f73fca005 (at 192.168.204.6@tcp) reconnecting [ 5247.510640] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5248.359382] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5251.051278] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5251.154314] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5253.278784] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5253.395435] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5254.646886] Lustre: Permanently deactivating lustre-MDT0001 [ 5254.655740] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5254.659595] Lustre: Skipped 2 previous similar messages [ 5263.374927] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5265.129660] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5265.133096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5265.138653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5270.497205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5270.506138] Lustre: Skipped 6 previous similar messages [ 5271.378391] Lustre: server umount lustre-MDT0000 complete [ 5273.614075] LustreError: 239126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781610255 with bad export cookie 4675081516922715018 [ 5273.620150] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5273.622735] LustreError: 239126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5273.846989] Lustre: server umount lustre-MDT0001 complete [ 5282.119977] Lustre: server umount lustre-OST0000 complete [ 5287.910743] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 07:44:28 (1781610268) [ 5293.338255] Lustre: server umount lustre-OST0001 complete [ 5296.913665] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 5302.961649] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5322.067221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5322.193054] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5322.198586] Lustre: Skipped 2 previous similar messages [ 5322.216618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5322.259358] Lustre: lustre-MDT0000: new disk, initializing [ 5322.306869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5322.323171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5324.220854] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5329.781948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5329.845867] Lustre: 247395:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5329.884933] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5329.887629] Lustre: Skipped 1 previous similar message [ 5329.927566] Lustre: lustre-MDT0001: new disk, initializing [ 5330.005502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5330.012990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5332.218823] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5337.214466] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5339.776520] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5343.195669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5343.339859] Lustre: lustre-OST0000: new disk, initializing [ 5343.342674] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5343.347383] Lustre: 249363:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5343.393630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5343.399595] Lustre: Skipped 1 previous similar message [ 5343.901882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5343.907940] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5343.959426] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5346.361956] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5351.748302] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5352.731677] LustreError: 247401:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5358.140353] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5358.259093] Lustre: lustre-OST0001: new disk, initializing [ 5358.261894] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5358.268353] Lustre: 250474:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5361.561973] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5362.743149] LustreError: 247401:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5362.756210] LustreError: 247403:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5364.775124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5364.779806] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5364.857159] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5366.916620] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5367.818408] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5372.767340] LustreError: 247403:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5374.951108] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5374.952274] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5374.956755] LustreError: Skipped 1 previous similar message [ 5374.963432] Lustre: Skipped 3 previous similar messages [ 5374.967853] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5380.066389] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5380.069222] Lustre: Skipped 3 previous similar messages [ 5385.188743] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5385.190905] Lustre: Skipped 1 previous similar message [ 5388.256362] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5388.318750] Lustre: server umount lustre-OST0001 complete [ 5390.306534] LustreError: 249346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5390.315093] LustreError: 249346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5395.430155] LustreError: 249348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5395.437922] LustreError: 249348:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5400.549559] LustreError: 249348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5400.558360] LustreError: 249348:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5405.665592] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5405.666359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5405.670317] LustreError: Skipped 1 previous similar message [ 5405.675541] Lustre: Skipped 1 previous similar message [ 5405.678372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5405.683032] Lustre: Skipped 2 previous similar messages [ 5407.646711] Lustre: server umount lustre-OST0000 complete [ 5410.784859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5410.790497] Lustre: Skipped 1 previous similar message [ 5415.625360] Lustre: server umount lustre-MDT0000 complete [ 5415.904937] LustreError: 247406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5415.912922] LustreError: 247406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5417.497966] LustreError: 247386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781610399 with bad export cookie 4675081516922717839 [ 5417.499512] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5417.508155] LustreError: 247386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5423.714527] Lustre: server umount lustre-MDT0001 complete [ 5429.636432] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5431.309717] Key type lgssc unregistered [ 5431.500063] LNet: 252506:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5431.504681] LNetError: 252506:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5431.517773] LNet: Removed LNI 192.168.204.106@tcp [ 5431.988815] Key type .llcrypt unregistered [ 5431.990439] Key type ._llcrypt unregistered [ 5453.137211] Key type ._llcrypt registered [ 5453.139456] Key type .llcrypt registered [ 5453.210401] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5453.838668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5453.853557] alg: No test for adler32 (adler32-zlib) [ 5454.790607] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 5454.920445] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 5456.519188] Key type lgssc registered [ 5457.051637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5460.723162] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5460.737897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5461.892686] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5461.909098] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5461.986301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5464.095466] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5468.591509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5468.664301] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5468.681397] Lustre: 254691:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5468.824568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5470.897719] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5474.555277] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5477.357760] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5480.894826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5481.026250] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5481.155245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5482.677466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 5484.548240] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5488.493322] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5534.177977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5534.184593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5534.196889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5534.202865] Lustre: Skipped 1 previous similar message [ 5537.023849] Lustre: server umount lustre-OST0000 complete [ 5539.510768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5546.775585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5547.004433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5550.506893] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5554.156615] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.169639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.169983] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5554.927150] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5589.987234] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5589.987865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5589.993519] LustreError: Skipped 2 previous similar messages [ 5589.994496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5590.011384] Lustre: Skipped 2 previous similar messages [ 5595.105042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5595.110921] Lustre: Skipped 2 previous similar messages [ 5595.872960] Lustre: server umount lustre-OST0000 complete [ 5600.224879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5600.232369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5600.235629] Lustre: Skipped 1 previous similar message [ 5604.084817] Lustre: server umount lustre-MDT0000 complete [ 5605.951315] LustreError: 254682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781610587 with bad export cookie 7499596936783575127 [ 5605.956671] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5605.962611] LustreError: 254682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5606.141082] Lustre: server umount lustre-MDT0001 complete [ 5611.381438] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5612.851901] Key type lgssc unregistered [ 5613.038965] LNet: 260107:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5613.045053] LNetError: 260107:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5613.054920] LNet: Removed LNI 192.168.204.106@tcp [ 5613.566352] Key type .llcrypt unregistered [ 5613.569249] Key type ._llcrypt unregistered [ 5626.622290] Key type ._llcrypt registered [ 5626.624163] Key type .llcrypt registered [ 5626.677743] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5627.174448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5627.201721] alg: No test for adler32 (adler32-zlib) [ 5628.153981] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 5628.306462] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 5629.935275] Key type lgssc registered [ 5630.572866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5634.833182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5634.846908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5636.137773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5638.580256] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5643.613671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5643.912072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5646.327599] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5650.255569] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5653.160191] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5656.494891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5656.691649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5659.835634] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5660.779329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5663.688566] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5681.633482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5681.643666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5681.654577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5684.706809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5684.708017] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5684.715067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5685.942626] Lustre: server umount lustre-OST0000 complete [ 5689.826842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5689.826905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5689.835714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5689.846909] Lustre: Skipped 2 previous similar messages [ 5693.899755] Lustre: server umount lustre-MDT0000 complete [ 5694.944806] LustreError: 261497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5694.953054] LustreError: 261497:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5695.824092] LustreError: 261482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781610677 with bad export cookie 1621247882990210574 [ 5695.825923] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5695.835295] LustreError: 261482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5696.013890] Lustre: server umount lustre-MDT0001 complete [ 5701.804485] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5703.525794] Key type lgssc unregistered [ 5703.728330] LNet: 265806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5703.732246] LNetError: 265806:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5704.806487] LNet: Removed LNI 192.168.204.106@tcp [ 5705.258987] Key type .llcrypt unregistered [ 5705.260982] Key type ._llcrypt unregistered [ 5715.122525] Key type ._llcrypt registered [ 5715.125164] Key type .llcrypt registered [ 5715.181579] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5715.821438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5715.831443] alg: No test for adler32 (adler32-zlib) [ 5716.788892] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 5716.941314] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 5718.575199] Key type lgssc registered [ 5719.259699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5723.805588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5723.815750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5725.094811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5727.389387] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5731.630350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5731.832106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5733.822178] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5737.410894] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5739.994677] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5743.724253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5743.943431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5747.319172] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5750.125739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5751.003881] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5761.507500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5761.510408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5761.516845] Lustre: Skipped 1 previous similar message [ 5761.517834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5761.520933] Lustre: Skipped 1 previous similar message [ 5763.206250] Lustre: server umount lustre-OST0000 complete [ 5766.630708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5766.637880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5766.640497] Lustre: Skipped 1 previous similar message [ 5771.578469] Lustre: server umount lustre-MDT0000 complete [ 5771.744877] LustreError: 267005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5771.755357] LustreError: 267005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5773.786316] LustreError: 266984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781610755 with bad export cookie 11135704515495604178 [ 5773.793141] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5773.972509] Lustre: server umount lustre-MDT0001 complete [ 5780.904287] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5782.723271] Key type lgssc unregistered [ 5782.915763] LNet: 270678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5782.919908] LNetError: 270678:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5782.929150] LNet: Removed LNI 192.168.204.106@tcp [ 5783.457546] Key type .llcrypt unregistered [ 5783.462593] Key type ._llcrypt unregistered [ 5798.084047] Key type ._llcrypt registered [ 5798.090436] Key type .llcrypt registered [ 5798.165919] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5798.797576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5798.862730] alg: No test for adler32 (adler32-zlib) [ 5799.791198] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 5799.947585] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 5801.575175] Key type lgssc registered [ 5802.268815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5807.016627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5807.041352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5808.342740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5811.118424] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5816.233241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5816.514980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5819.088259] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5823.442514] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5826.824889] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5830.915855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5831.103489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5834.156726] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5834.642817] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5839.043087] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5854.688940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5854.698044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5854.707862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5856.738257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5856.740984] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5856.753766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5859.839562] Lustre: server umount lustre-OST0000 complete [ 5862.880343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5862.887587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5862.897450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5862.901137] Lustre: Skipped 1 previous similar message [ 5866.976650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5866.976890] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5872.098106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5872.101728] Lustre: Skipped 2 previous similar messages [ 5876.191172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5876.324654] Lustre: server umount lustre-MDT0000 complete [ 5878.416055] LustreError: 272050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781610860 with bad export cookie 1676622756320193562 [ 5878.422174] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5878.425569] LustreError: 272050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5878.614437] Lustre: server umount lustre-MDT0001 complete [ 5883.963883] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5885.685475] Key type lgssc unregistered [ 5885.894341] LNet: 276374:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5885.898389] LNetError: 276374:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5885.907785] LNet: Removed LNI 192.168.204.106@tcp [ 5886.432380] Key type .llcrypt unregistered [ 5886.433895] Key type ._llcrypt unregistered [ 5895.705985] Key type ._llcrypt registered [ 5895.707521] Key type .llcrypt registered [ 5895.774235] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5896.379637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5896.392659] alg: No test for adler32 (adler32-zlib) [ 5897.289909] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 5897.407444] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 5899.007106] Key type lgssc registered [ 5899.590550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5903.682537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5903.696175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5904.999892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5907.237498] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5912.025431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5912.250969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5914.507109] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5918.524733] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5921.773264] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5925.942629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5926.228779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5929.322995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 5930.074612] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5935.065595] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5947.873767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5947.877519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5947.880912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5947.898953] Lustre: Skipped 1 previous similar message [ 5947.909950] Lustre: Skipped 1 previous similar message [ 5952.274410] Lustre: server umount lustre-OST0000 complete [ 5958.113319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5958.115749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5958.124466] Lustre: Skipped 1 previous similar message [ 5960.605791] Lustre: server umount lustre-MDT0000 complete [ 5962.817340] LustreError: 277553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781610944 with bad export cookie 12726729381866681440 [ 5962.820290] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5962.824380] LustreError: 277553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5963.064086] Lustre: server umount lustre-MDT0001 complete [ 5969.607730] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5971.390479] Key type lgssc unregistered [ 5971.605801] LNet: 281284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5971.614990] LNetError: 281284:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5971.632515] LNet: Removed LNI 192.168.204.106@tcp [ 5972.152224] Key type .llcrypt unregistered [ 5972.153918] Key type ._llcrypt unregistered [ 5978.549960] Key type ._llcrypt registered [ 5978.551791] Key type .llcrypt registered [ 5978.608206] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5988.172821] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 5996.837239] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5997.317174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5997.385121] alg: No test for adler32 (adler32-zlib) [ 5998.320269] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 5998.451894] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 6000.072382] Key type lgssc registered [ 6000.635455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6019.577741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6019.586080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6020.722537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6020.755476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6020.825565] Lustre: lustre-MDT0000: new disk, initializing [ 6020.878507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6020.897621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6023.215108] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6029.373822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6029.444454] Lustre: 284810:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6029.478724] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6029.481689] Lustre: Skipped 1 previous similar message [ 6029.526041] Lustre: lustre-MDT0001: new disk, initializing [ 6029.576469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6029.595849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6029.601585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6031.681332] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6037.098424] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6040.158787] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6043.727859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6043.867387] Lustre: lustre-OST0000: new disk, initializing [ 6043.870748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6043.875815] Lustre: 286778:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6043.933808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6047.056749] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6050.827652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6050.833918] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6050.917375] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6052.330588] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6054.609078] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6054.723551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6056.924811] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6057.053673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6061.024742] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6061.025543] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6061.028038] LustreError: Skipped 1 previous similar message [ 6061.028754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6061.041540] Lustre: Skipped 1 previous similar message [ 6061.058768] Lustre: Skipped 1 previous similar message [ 6064.256755] Lustre: server umount lustre-OST0000 complete [ 6070.753993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6070.760939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6070.770930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6072.475368] Lustre: server umount lustre-MDT0000 complete [ 6074.737401] LustreError: 284801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781611056 with bad export cookie 7542733796506883643 [ 6074.740256] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6074.743440] LustreError: 284801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6074.964888] Lustre: server umount lustre-MDT0001 complete [ 6079.617311] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 07:57:39 (1781611059) [ 6083.216313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6083.486723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6085.866388] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6088.673127] LustreError: 289187:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6088.681122] LustreError: 289187:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6090.570975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6090.776289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6093.025549] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6097.122968] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6100.101775] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6104.112268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6104.325984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6107.611790] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6111.871423] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6124.000898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6124.005259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6124.011696] Lustre: Skipped 1 previous similar message [ 6124.014794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6124.019286] Lustre: Skipped 2 previous similar messages [ 6126.051498] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6126.052533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6126.058170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6126.078668] Lustre: Skipped 1 previous similar message [ 6127.988534] Lustre: server umount lustre-OST0000 complete [ 6131.170991] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6131.189156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6131.192069] Lustre: Skipped 1 previous similar message [ 6136.123188] Lustre: server umount lustre-MDT0000 complete [ 6136.288486] LustreError: 289187:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6136.296911] LustreError: 289187:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6137.977453] LustreError: 289168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781611119 with bad export cookie 7542733796506884518 [ 6137.981250] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6137.985391] LustreError: 289168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6138.188373] Lustre: server umount lustre-MDT0001 complete [ 6143.784166] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 6145.349648] Key type lgssc unregistered [ 6145.508966] LNet: 292718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6145.514564] LNetError: 292718:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6145.523920] LNet: Removed LNI 192.168.204.106@tcp [ 6145.985155] Key type .llcrypt unregistered [ 6145.987989] Key type ._llcrypt unregistered [ 6160.323403] Key type ._llcrypt registered [ 6160.325230] Key type .llcrypt registered [ 6160.397444] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 6168.842804] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6169.349974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6169.411661] alg: No test for adler32 (adler32-zlib) [ 6170.325572] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 6170.454223] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 6172.079515] Key type lgssc registered [ 6172.723434] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6198.243464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6198.259737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6199.413253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6199.435385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6199.490393] Lustre: lustre-MDT0000: new disk, initializing [ 6199.532996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6199.545846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6202.157760] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6208.703289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6208.777944] Lustre: 296519:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6208.810226] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6208.813214] Lustre: Skipped 1 previous similar message [ 6208.857446] Lustre: lustre-MDT0001: new disk, initializing [ 6208.919871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6208.954749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6208.963352] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6211.334075] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6216.543078] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6219.429594] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6222.867545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6223.007379] Lustre: lustre-OST2710: new disk, initializing [ 6223.012609] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6223.018475] Lustre: 298487:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6223.062876] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6226.111690] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6229.025603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6229.031712] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6229.086748] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6231.437399] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 6235.137400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6235.236084] Lustre: lustre-OST03e8: new disk, initializing [ 6235.239643] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6235.244678] Lustre: 299548:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6235.299168] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6236.062894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6236.067529] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6236.129093] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6238.483836] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6242.556577] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 6246.407451] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6246.531174] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6248.697169] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6248.810418] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6250.889520] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6251.014157] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6253.174320] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6253.286631] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6256.611604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6256.617238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6256.618479] Lustre: Skipped 3 previous similar messages [ 6256.623467] Lustre: Skipped 2 previous similar messages [ 6261.332849] Lustre: server umount lustre-MDT0000 complete [ 6263.304181] LustreError: 296511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781611245 with bad export cookie 5312600517638849688 [ 6263.308030] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6263.313546] LustreError: 296511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6263.536223] Lustre: server umount lustre-MDT0001 complete [ 6276.140626] Lustre: server umount lustre-OST2710 complete [ 6288.655620] Lustre: server umount lustre-OST03e8 complete [ 6298.987588] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 6304.026660] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6322.085847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6322.228358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6322.255064] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6322.300157] Lustre: lustre-MDT0000: new disk, initializing [ 6322.348322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6322.360508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6324.209795] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6329.743355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6329.809941] Lustre: 304803:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6329.964697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6331.942271] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6336.701203] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6339.266213] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6342.748864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6342.872279] Lustre: lustre-OST0000: new disk, initializing [ 6342.874258] Lustre: Skipped 1 previous similar message [ 6342.876146] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6342.879407] Lustre: Skipped 2 previous similar messages [ 6342.882496] Lustre: 306769:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6342.917420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6342.921346] Lustre: Skipped 1 previous similar message [ 6345.469581] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6350.365240] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6351.363702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6351.370701] Lustre: Skipped 1 previous similar message [ 6351.375154] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6351.423737] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6352.409522] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6352.533310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6354.698181] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6354.812509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6356.452868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6356.453100] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6356.453982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6356.461553] Lustre: Skipped 1 previous similar message [ 6356.471650] LustreError: Skipped 1 previous similar message [ 6357.860155] Lustre: server umount lustre-OST0000 complete [ 6360.543959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6360.548167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6360.554608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6360.558423] Lustre: Skipped 1 previous similar message [ 6365.752810] Lustre: server umount lustre-MDT0000 complete [ 6366.689026] LustreError: 304813:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6366.698786] LustreError: 304813:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6367.817671] LustreError: 306768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781611349 with bad export cookie 5312600517638851368 [ 6367.820026] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6367.829197] LustreError: 306768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6367.990910] Lustre: server umount lustre-MDT0001 complete [ 6372.463744] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 08:02:32 (1781611352) [ 6373.196414] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6374.075461] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 08:02:34 (1781611354) [ 6375.848396] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6389.783309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6389.921780] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6389.936545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6390.026848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6391.906219] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6396.027654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6396.095679] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6396.112451] Lustre: 310621:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6398.299860] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6415.909508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6416.004585] Lustre: Denying initial registration attempt from nid 192.168.204.106@tcp, specified as failover [ 6416.011671] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6416.016903] LustreError: 312073:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 6416.021199] LustreError: 312073:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 6416.024665] LustreError: 312073:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6416.063327] Lustre: server umount lustre-OST0000 complete [ 6416.065172] LustreError: 312073:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6421.984629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6421.989204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6421.995109] Lustre: Skipped 1 previous similar message [ 6421.998213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6422.002078] Lustre: Skipped 2 previous similar messages [ 6425.408283] Lustre: server umount lustre-MDT0000 complete [ 6427.368254] LustreError: 311362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781611409 with bad export cookie 5312600517638852236 [ 6427.369180] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6427.378045] LustreError: 311362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6427.524980] Lustre: server umount lustre-MDT0001 complete [ 6432.476160] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 6433.797213] Key type lgssc unregistered [ 6433.963982] LNet: 313194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6433.969535] LNetError: 313194:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6433.984788] LNet: Removed LNI 192.168.204.106@tcp [ 6434.389708] Key type .llcrypt unregistered [ 6434.391249] Key type ._llcrypt unregistered [ 6439.850976] Key type ._llcrypt registered [ 6439.852467] Key type .llcrypt registered [ 6439.906520] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6440.326867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6440.382533] alg: No test for adler32 (adler32-zlib) [ 6441.320623] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 6441.445853] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 6443.047378] Key type lgssc registered [ 6443.561795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6473.771439] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6478.173781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6478.188276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6479.375988] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6479.401620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6479.501112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6481.458560] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6485.673238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6485.735333] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6485.750205] Lustre: 315940:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6485.890121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6487.884200] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6491.536951] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6494.082774] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6510.381946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6510.494160] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6510.604173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6513.388579] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6516.910152] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6521.829142] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6521.829277] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6521.837059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6521.840290] LustreError: Skipped 1 previous similar message [ 6521.845248] Lustre: Skipped 1 previous similar message [ 6524.060365] Lustre: server umount lustre-OST0000 complete [ 6526.946804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6526.949452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6526.950515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6526.958473] Lustre: Skipped 2 previous similar messages [ 6532.068441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6532.074269] Lustre: Skipped 2 previous similar messages [ 6532.220303] Lustre: server umount lustre-MDT0000 complete [ 6533.948861] LustreError: 315934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781611515 with bad export cookie 18250343736701464696 [ 6533.951619] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6533.956089] LustreError: 315934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6534.113304] Lustre: server umount lustre-MDT0001 complete [ 6538.800622] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 6540.276059] Key type lgssc unregistered [ 6540.455559] LNet: 319508:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6540.460478] LNetError: 319508:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6540.471692] LNet: Removed LNI 192.168.204.106@tcp [ 6540.915386] Key type .llcrypt unregistered [ 6540.917055] Key type ._llcrypt unregistered [ 6553.950443] Key type ._llcrypt registered [ 6553.952410] Key type .llcrypt registered [ 6554.016883] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6554.526610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6554.542226] alg: No test for adler32 (adler32-zlib) [ 6555.466835] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 6555.624472] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 6557.231221] Key type lgssc registered [ 6557.916487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6562.388339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6562.414991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6563.683431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6566.010921] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6570.579201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6570.827252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6573.285384] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6577.396481] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6580.485762] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6586.533838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6586.727615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6589.837232] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6594.058149] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6601.185155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6601.189048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6601.194485] Lustre: Skipped 2 previous similar messages [ 6601.201542] Lustre: Skipped 2 previous similar messages [ 6601.789772] Lustre: server umount lustre-MDT0000 complete [ 6603.850828] LustreError: 320883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781611585 with bad export cookie 18124084600423732953 [ 6603.854121] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6603.857593] LustreError: 320883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6606.305348] LustreError: 320899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6606.305413] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6606.315472] LustreError: 320899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6606.337252] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6606.341064] Lustre: Skipped 1 previous similar message [ 6608.353510] LustreError: 320900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6608.354090] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6608.367382] LustreError: 320900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6608.381764] Lustre: Skipped 1 previous similar message [ 6610.105752] Lustre: server umount lustre-MDT0001 complete [ 6613.689033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6618.471580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6628.835487] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfb85aeeb276272d9 [ 6628.840441] Lustre: MGC192.168.204.106@tcp: Connection restored to 0@lo (at 0@lo) [ 6629.012729] LustreError: 322733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6629.069160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6629.110743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 6631.409967] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6633.185136] LustreError: 324492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6633.196606] LustreError: 324492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6636.067948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6636.315710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6638.377749] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6638.505267] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6642.597294] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6643.427974] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6646.061677] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6647.963496] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 1 sec [ 6651.873889] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6651.880741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6651.894143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6653.927252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6656.322853] Lustre: server umount lustre-OST0000 complete [ 6659.044844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6659.049983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6659.060257] Lustre: Skipped 1 previous similar message [ 6659.068298] Lustre: Skipped 3 previous similar messages [ 6664.671134] Lustre: server umount lustre-MDT0000 complete [ 6666.790458] LustreError: 320883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781611648 with bad export cookie 18124084600423740121 [ 6666.793408] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6666.797060] LustreError: 320883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6666.967701] Lustre: server umount lustre-MDT0001 complete [ 6672.644828] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 6674.232475] Key type lgssc unregistered [ 6674.401431] LNet: 327162:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6674.410741] LNetError: 327162:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6674.421965] LNet: Removed LNI 192.168.204.106@tcp [ 6674.897797] Key type .llcrypt unregistered [ 6674.899434] Key type ._llcrypt unregistered [ 6687.447603] Key type ._llcrypt registered [ 6687.449553] Key type .llcrypt registered [ 6687.497864] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6688.044843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6688.056217] alg: No test for adler32 (adler32-zlib) [ 6688.988110] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 6689.118630] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 6690.735217] Key type lgssc registered [ 6691.382720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6695.963872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6695.986115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6697.270378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6699.713598] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6704.605512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6704.825427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6707.063469] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6711.085032] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6714.237137] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6720.484726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6720.485422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6720.486967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6720.507269] Lustre: Skipped 1 previous similar message [ 6722.319492] Lustre: server umount lustre-MDT0000 complete [ 6724.266818] LustreError: 328537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781611706 with bad export cookie 1769589818008197707 [ 6724.270557] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6724.272658] LustreError: 328537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6724.445021] Lustre: server umount lustre-MDT0001 complete [ 6732.138117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6732.346614] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6732.372640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6732.518076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6734.521370] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6738.511265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6738.571226] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6738.590735] Lustre: 331141:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6738.734688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6740.616674] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6744.138506] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6746.794534] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6752.424423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6752.516783] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6752.523846] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6752.528878] LustreError: 333024:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 6752.534526] LustreError: 333024:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 6752.537484] LustreError: 333024:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6752.572375] Lustre: server umount lustre-OST0000 complete [ 6752.574495] LustreError: 333024:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6758.565659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6758.672140] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6758.784357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6761.649428] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6765.335996] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6766.118345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 6769.253340] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6769.320602] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6769.394684] Lustre: lustre-OST0001: new disk, initializing [ 6769.397585] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6769.401132] Lustre: 334565:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6769.456864] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6772.342271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6772.356589] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6772.440605] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6772.916247] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6776.718669] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6782.951563] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6782.953364] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6782.957079] LustreError: Skipped 1 previous similar message [ 6782.966763] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6782.971428] Lustre: Skipped 2 previous similar messages [ 6784.529755] Lustre: server umount lustre-OST0001 complete [ 6786.528100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6786.531205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6786.532439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6786.536040] Lustre: Skipped 2 previous similar messages [ 6788.070442] LustreError: 333491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6788.083165] LustreError: 333491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6792.620553] Lustre: server umount lustre-OST0000 complete [ 6795.237571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6795.241851] LustreError: Skipped 1 previous similar message [ 6795.244687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6795.252977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6795.255360] Lustre: Skipped 3 previous similar messages [ 6800.769812] Lustre: server umount lustre-MDT0000 complete [ 6802.572700] LustreError: 331133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781611784 with bad export cookie 1769589818008198260 [ 6802.573947] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6802.585916] LustreError: 331133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6802.745130] Lustre: server umount lustre-MDT0001 complete [ 6807.543955] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 6808.919673] Key type lgssc unregistered [ 6809.079670] LNet: 336520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6809.083671] LNetError: 336520:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6809.093580] LNet: Removed LNI 192.168.204.106@tcp [ 6809.495191] Key type .llcrypt unregistered [ 6809.497292] Key type ._llcrypt unregistered [ 6849.881859] Key type ._llcrypt registered [ 6849.883618] Key type .llcrypt registered [ 6849.948247] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 6857.976276] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6858.517623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6858.531679] alg: No test for adler32 (adler32-zlib) [ 6859.447588] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 6859.571201] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 6861.183160] Key type lgssc registered [ 6861.771082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6880.492295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6880.506982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6881.660365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6881.681734] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6881.730534] Lustre: lustre-MDT0000: new disk, initializing [ 6881.784778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6881.797114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6884.069473] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6889.965734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6890.022883] Lustre: 341426:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6890.060824] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6890.064389] Lustre: Skipped 1 previous similar message [ 6890.110763] Lustre: lustre-MDT0001: new disk, initializing [ 6890.167215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6890.200758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6890.205850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6892.291262] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6897.300680] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6899.828694] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6903.059289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6903.197396] Lustre: lustre-OST0000: new disk, initializing [ 6903.199929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6903.204609] Lustre: 343393:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6903.241352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6906.119611] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6911.429715] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6912.024381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6912.030860] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6912.099574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6913.461475] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6913.582790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6915.622398] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6915.734538] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6917.093459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6917.093539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6917.101798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6917.104476] LustreError: Skipped 1 previous similar message [ 6917.110510] Lustre: Skipped 1 previous similar message [ 6918.696158] Lustre: server umount lustre-OST0000 complete [ 6921.183953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6921.186627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6921.191429] Lustre: Skipped 1 previous similar message [ 6921.193885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6922.209411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6922.209597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6922.223329] Lustre: Skipped 1 previous similar message [ 6926.517332] Lustre: server umount lustre-MDT0000 complete [ 6928.119372] LustreError: 341418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781611909 with bad export cookie 15547661308415969488 [ 6928.120581] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6928.124932] LustreError: 341418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6928.274443] Lustre: server umount lustre-MDT0001 complete [ 6932.109619] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 08:11:52 (1781611912) [ 6935.551673] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 08:11:55 (1781611915) [ 6939.976798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6940.210365] LustreError: 346143:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6940.219394] LustreError: 346143:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6940.246671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6942.326287] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6945.250401] LustreError: 346144:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6946.469733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6946.688533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6948.902726] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6952.559786] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6955.363354] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6958.705464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6958.894546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6961.743725] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6965.403742] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6975.223374] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6976.174723] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6977.051884] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6977.946261] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6978.905797] Lustre: Failing over lustre-MDT0000 [ 6979.092189] Lustre: server umount lustre-MDT0000 complete [ 6981.601614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6981.602858] LustreError: 348628:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6981.612068] Lustre: Skipped 2 previous similar messages [ 6981.630663] LustreError: 348628:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6986.721458] LustreError: 346143:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6986.729315] LustreError: 346143:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6991.840154] LustreError: 346144:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6991.851607] LustreError: 346144:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6993.788444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6993.863318] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6994.032614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6994.065028] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6996.384597] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6998.399919] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6999.527685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6999.545380] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6999.571168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7002.384967] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7003.293974] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7005.600104] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7006.647599] Lustre: Failing over lustre-MDT0000 [ 7006.798942] Lustre: server umount lustre-MDT0000 complete [ 7008.658119] LustreError: 348628:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7008.670571] LustreError: 348628:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7009.761773] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7009.761908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7021.255887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7021.317168] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7021.504409] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7023.822575] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7023.997436] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7026.663750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7026.666511] Lustre: Skipped 2 previous similar messages [ 7026.682203] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7026.702421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7029.261084] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7029.982913] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7036.898181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7036.898905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7036.900275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7036.904360] Lustre: Skipped 4 previous similar messages [ 7038.779117] Lustre: server umount lustre-MDT0000 complete [ 7040.639722] LustreError: 346125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781612022 with bad export cookie 15547661308415972169 [ 7040.643149] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7040.644929] LustreError: 346125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7040.830767] Lustre: server umount lustre-MDT0001 complete [ 7053.093387] Lustre: server umount lustre-OST0000 complete [ 7060.266524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7060.511468] LustreError: 352473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7060.523734] LustreError: 352473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 7060.551457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7060.555698] Lustre: Skipped 1 previous similar message [ 7062.590810] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7066.867828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7069.459438] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7073.219449] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7075.700882] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7078.837077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7081.828644] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7085.612719] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7086.184239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7101.922771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7101.924226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7101.926664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7101.926671] Lustre: Skipped 2 previous similar messages [ 7101.934118] LustreError: Skipped 1 previous similar message [ 7101.941200] Lustre: Skipped 1 previous similar message [ 7104.051576] Lustre: server umount lustre-OST0000 complete [ 7112.160210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7112.165664] Lustre: Skipped 4 previous similar messages [ 7120.351181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7120.453358] Lustre: server umount lustre-MDT0000 complete [ 7121.376168] LustreError: 352473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7121.383756] LustreError: 352473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7122.113425] LustreError: 352453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781612103 with bad export cookie 15547661308415972918 [ 7122.116214] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7122.121697] LustreError: 352453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7122.298059] Lustre: server umount lustre-MDT0001 complete [ 7127.245554] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7128.712304] Key type lgssc unregistered [ 7128.868856] LNet: 356023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7128.871790] LNetError: 356023:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7128.880332] LNet: Removed LNI 192.168.204.106@tcp [ 7129.255334] Key type .llcrypt unregistered [ 7129.256596] Key type ._llcrypt unregistered [ 7141.962498] Key type ._llcrypt registered [ 7141.964121] Key type .llcrypt registered [ 7142.018876] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7142.418246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7142.427781] alg: No test for adler32 (adler32-zlib) [ 7143.346492] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 7143.485041] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 7145.095219] Key type lgssc registered [ 7145.683086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7149.635532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7149.657675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7150.918309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7153.164313] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7157.052659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7157.221027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7158.917864] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7162.329601] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7164.821578] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7168.087498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7168.264920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7170.952914] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7173.997611] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7175.466543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7193.057562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7193.057664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7193.058921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7193.064717] Lustre: Skipped 1 previous similar message [ 7194.995152] Lustre: server umount lustre-MDT0000 complete [ 7196.494586] LustreError: 357498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781612178 with bad export cookie 10142712803225051475 [ 7196.498611] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7196.499378] LustreError: 357498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7196.621761] Lustre: server umount lustre-MDT0001 complete [ 7208.438594] Lustre: server umount lustre-OST0000 complete [ 7214.528535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7214.768869] LustreError: 361243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7214.780595] LustreError: 361243:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7214.802877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7216.734203] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7220.192376] LustreError: 361244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7220.979640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7221.179242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7223.156254] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7226.952283] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7229.725554] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7233.221127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7233.428594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7236.295183] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7239.596976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7239.623244] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7251.941122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7251.941410] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7251.943191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7251.943200] Lustre: Skipped 2 previous similar messages [ 7251.957983] Lustre: Skipped 2 previous similar messages [ 7253.378883] Lustre: server umount lustre-MDT0000 complete [ 7255.296385] LustreError: 361979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781612237 with bad export cookie 10142712803225052574 [ 7255.299510] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7255.303973] LustreError: 361979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7255.473944] Lustre: server umount lustre-MDT0001 complete [ 7267.636623] Lustre: server umount lustre-OST0000 complete [ 7274.645961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7274.903563] LustreError: 365149:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7274.950935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7276.764237] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7280.097071] LustreError: 365150:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7280.458790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7282.433431] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7285.542884] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7287.827311] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7290.951653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7293.741971] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7296.958569] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7299.369129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7304.672924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7304.678180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7304.684980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7304.688189] Lustre: Skipped 2 previous similar messages [ 7306.146141] Lustre: server umount lustre-OST0000 complete [ 7310.007683] Lustre: server umount lustre-MDT0000 complete [ 7310.817752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7310.818874] LustreError: 365146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7310.824799] Lustre: Skipped 1 previous similar message [ 7312.293533] LustreError: 365883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781612294 with bad export cookie 10142712803225053575 [ 7312.295702] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7312.301158] LustreError: 365883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7314.622396] Lustre: server umount lustre-MDT0001 complete [ 7326.359517] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7329.130898] Key type lgssc unregistered [ 7329.364744] LNet: 368676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7329.369693] LNetError: 368676:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7330.415403] LNet: Removed LNI 192.168.204.106@tcp [ 7331.445927] Key type .llcrypt unregistered [ 7331.447986] Key type ._llcrypt unregistered [ 7357.643462] Key type ._llcrypt registered [ 7357.645614] Key type .llcrypt registered [ 7357.719722] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7358.406060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7358.434498] alg: No test for adler32 (adler32-zlib) [ 7359.380963] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 7359.531457] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 7361.167138] Key type lgssc registered [ 7361.922911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7366.476551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7366.490482] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7366.494460] LustreError: 370118:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 7366.506449] LustreError: 370118:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 7366.511080] LustreError: 370118:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 7366.515905] LustreError: 370118:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 7366.522016] LustreError: 370118:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7370.686107] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7370.694861] LustreError: 370514:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 7370.710331] LustreError: 370514:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 7370.715583] LustreError: 370514:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 7370.722270] LustreError: 370514:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 7370.726203] LustreError: 370514:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7381.002747] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7382.540684] Key type lgssc unregistered [ 7382.718777] LNet: 371516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7382.723492] LNetError: 371516:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7382.738435] LNet: Removed LNI 192.168.204.106@tcp [ 7383.115881] Key type .llcrypt unregistered [ 7383.118411] Key type ._llcrypt unregistered [ 7393.558643] Key type ._llcrypt registered [ 7393.560465] Key type .llcrypt registered [ 7393.604214] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 7403.521608] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7404.120819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7404.147396] alg: No test for adler32 (adler32-zlib) [ 7405.052497] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 7405.198042] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 7406.815187] Key type lgssc registered [ 7407.519287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7428.190793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7428.210174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7429.371304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7429.396127] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7429.462962] Lustre: lustre-MDT0000: new disk, initializing [ 7429.511445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7429.522532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7432.231157] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7438.678884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7438.760366] Lustre: 374648:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7438.807456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7438.811884] Lustre: Skipped 1 previous similar message [ 7438.856548] Lustre: lustre-MDT0001: new disk, initializing [ 7438.925236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7438.948876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7438.955356] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7441.522981] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7447.531511] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7450.543677] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7454.507058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7454.679376] Lustre: lustre-OST0000: new disk, initializing [ 7454.681831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7454.685390] Lustre: 376613:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7454.744057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7455.827129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7455.837055] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7455.919578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7458.416827] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7463.574864] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7465.613837] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7465.718523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7467.975281] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7468.090640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7471.587659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7471.587983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7471.588822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7471.593819] Lustre: Skipped 1 previous similar message [ 7471.601691] LustreError: Skipped 1 previous similar message [ 7475.322467] Lustre: server umount lustre-OST0000 complete [ 7479.776479] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7479.784294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7479.796253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7479.800208] Lustre: Skipped 1 previous similar message [ 7481.824929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7481.825603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7483.639067] Lustre: server umount lustre-MDT0000 complete [ 7485.952820] LustreError: 375443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781612467 with bad export cookie 8448429653104203251 [ 7485.955321] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7485.959171] LustreError: 375443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7486.191513] Lustre: server umount lustre-MDT0001 complete [ 7491.331212] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 7492.364938] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 08:21:12 (1781612472) [ 7496.311632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7496.600033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7499.339119] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7501.793375] LustreError: 379071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7501.803695] LustreError: 379071:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7504.549635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7504.765883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7507.325223] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7511.724755] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7514.787711] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7518.760872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7518.973227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7522.650900] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7526.351116] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7529.830156] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7529.932327] Lustre: lustre-OST0001: new disk, initializing [ 7529.935728] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7529.939637] Lustre: 381902:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7533.385196] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7536.671449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7536.678269] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7536.744683] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7538.404611] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7541.732311] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7541.736186] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7541.738194] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7541.742231] Lustre: Skipped 1 previous similar message [ 7541.748803] Lustre: Skipped 2 previous similar messages [ 7546.773323] Lustre: server umount lustre-OST0001 complete [ 7546.856211] LustreError: 382485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7551.975348] LustreError: 380888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7551.983303] LustreError: 380888:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7557.094835] LustreError: 380890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7557.102551] LustreError: 380890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7560.161363] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7560.167498] LustreError: Skipped 1 previous similar message [ 7560.170772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7560.178105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7560.181328] Lustre: Skipped 1 previous similar message [ 7562.209953] LustreError: 380891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7562.220078] LustreError: 380891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7566.003805] Lustre: server umount lustre-OST0000 complete [ 7571.425118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7571.429162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7571.436843] Lustre: Skipped 1 previous similar message [ 7571.439754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7571.443089] Lustre: Skipped 2 previous similar messages [ 7574.142141] Lustre: server umount lustre-MDT0000 complete [ 7576.027232] LustreError: 379052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781612557 with bad export cookie 8448429653104204133 [ 7576.031506] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7576.036555] LustreError: 379052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7590.367193] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7590.518793] Lustre: server umount lustre-MDT0001 complete [ 7595.520913] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7596.902231] Key type lgssc unregistered [ 7597.058459] LNet: 383889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7597.062546] LNetError: 383889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7598.117555] LNet: Removed LNI 192.168.204.106@tcp [ 7598.526239] Key type .llcrypt unregistered [ 7598.527944] Key type ._llcrypt unregistered [ 7622.426091] Key type ._llcrypt registered [ 7622.427707] Key type .llcrypt registered [ 7622.474088] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7623.021610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7623.030233] alg: No test for adler32 (adler32-zlib) [ 7623.949701] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 7624.077380] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 7625.687176] Key type lgssc registered [ 7626.290384] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7630.034905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7630.043809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7631.180647] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7631.196335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7631.287784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7633.032423] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7636.746495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7636.804272] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7636.823291] Lustre: 386661:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7636.954083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7638.711313] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7642.035623] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7644.734641] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7647.200374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7647.204395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7647.211149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7648.736928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7648.738405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7651.798985] Lustre: server umount lustre-MDT0000 complete [ 7653.295574] LustreError: 386653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781612635 with bad export cookie 11501955339572247775 [ 7653.298566] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7653.301420] LustreError: 386653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7653.427935] Lustre: server umount lustre-MDT0001 complete [ 7657.221485] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 08:23:57 (1781612637) [ 7660.150625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7660.394790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7662.067776] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7665.633058] LustreError: 389174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7665.640022] LustreError: 389174:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7665.806786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7665.976532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7667.671922] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7670.543757] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7672.704441] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7675.609879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7675.748865] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7675.844388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7678.295950] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7681.597590] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7686.993379] Lustre: Permanently deactivating lustre-OST0000 [ 7686.999663] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7687.535348] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7689.245837] LustreError: 392013:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 7689.250136] LustreError: 392013:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 7689.253570] LustreError: 392013:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7690.283499] LustreError: 392061:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 7690.289680] LustreError: 392061:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7697.346239] Lustre: server umount lustre-OST0000 complete [ 7698.799387] LustreError: 392311:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 7698.804690] LustreError: 392311:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7701.984082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7701.988462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7701.997069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7702.005415] Lustre: Skipped 1 previous similar message [ 7707.616754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7707.619993] Lustre: Skipped 3 previous similar messages [ 7712.737092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7712.740934] Lustre: Skipped 1 previous similar message [ 7714.271122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7714.350366] Lustre: server umount lustre-MDT0000 complete [ 7716.158182] LustreError: 389155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781612698 with bad export cookie 11501955339572248342 [ 7716.159357] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7716.169109] LustreError: 389155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7726.647944] Lustre: server umount lustre-MDT0001 complete [ 7731.106033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7733.423684] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7734.812878] LustreError: 393469:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 7737.778246] Lustre: 393807:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.106@tcp [ 7738.841202] Lustre: server umount MGS complete [ 7744.571502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7744.833748] LustreError: 394530:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7744.841514] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7744.848870] Lustre: Skipped 1 previous similar message [ 7744.877310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7746.900337] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7750.112618] LustreError: 394531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7751.031191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7751.174731] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7753.076787] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7756.633962] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7759.081708] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7760.019685] Lustre: Permanently reactivating lustre-OST0000 [ 7760.029207] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7760.034340] Lustre: Skipped 2 previous similar messages [ 7781.344081] LustreError: 396167:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 7784.240987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7784.399140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7784.401723] Lustre: Skipped 1 previous similar message [ 7787.523661] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7791.317657] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7806.949386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7806.949620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7806.959442] Lustre: Skipped 1 previous similar message [ 7806.962644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7806.967844] Lustre: Skipped 1 previous similar message [ 7810.340565] Lustre: server umount lustre-OST0000 complete [ 7812.576089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7812.582561] LustreError: Skipped 1 previous similar message [ 7812.586608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7812.597530] Lustre: Skipped 1 previous similar message [ 7826.399146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7826.400482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7826.407528] Lustre: Skipped 6 previous similar messages [ 7826.476490] Lustre: server umount lustre-MDT0000 complete [ 7828.167396] LustreError: 394512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781612810 with bad export cookie 11501955339572249364 [ 7828.169032] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7828.173798] LustreError: 394512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7828.348184] Lustre: server umount lustre-MDT0001 complete [ 7833.009484] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7834.348139] Key type lgssc unregistered [ 7834.494444] LNet: 398958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7834.498611] LNetError: 398958:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7834.513515] LNet: Removed LNI 192.168.204.106@tcp [ 7834.867384] Key type .llcrypt unregistered [ 7834.868979] Key type ._llcrypt unregistered [ 7842.516494] Key type ._llcrypt registered [ 7842.518405] Key type .llcrypt registered [ 7842.579491] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 7850.075793] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7850.571633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7850.606290] alg: No test for adler32 (adler32-zlib) [ 7851.500097] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 7851.595812] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 7853.183233] Key type lgssc registered [ 7853.691833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7868.445827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7868.453936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7869.578835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7869.594103] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7869.636423] Lustre: lustre-MDT0000: new disk, initializing [ 7869.668136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7869.677660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7871.301160] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7875.970416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7876.028159] Lustre: 402087:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7876.064248] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7876.067556] Lustre: Skipped 1 previous similar message [ 7876.107286] Lustre: lustre-MDT0001: new disk, initializing [ 7876.150713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7876.173935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7876.179519] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7877.781496] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7882.053201] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7884.243589] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7886.872046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7886.987585] Lustre: lustre-OST0000: new disk, initializing [ 7886.990416] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7886.995522] Lustre: 404057:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7887.033240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7889.182335] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7892.486272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7892.492452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7892.546565] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7893.587309] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7895.342576] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7895.439157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7897.106560] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7897.199988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7902.691318] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7902.692168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7902.693313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7902.694632] LustreError: Skipped 1 previous similar message [ 7902.706043] Lustre: Skipped 1 previous similar message [ 7904.093289] Lustre: server umount lustre-OST0000 complete [ 7906.784247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7906.787361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7906.793332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7906.795698] Lustre: Skipped 1 previous similar message [ 7907.808219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7907.808919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7911.713411] Lustre: server umount lustre-MDT0000 complete [ 7913.211433] LustreError: 404056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781612895 with bad export cookie 14896557052993715743 [ 7913.214502] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7913.216984] LustreError: 404056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7913.347604] Lustre: server umount lustre-MDT0001 complete [ 7916.638449] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7917.307855] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 08:28:17 (1781612897) [ 7919.910538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7920.076783] LustreError: 406514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7920.084552] LustreError: 406514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7920.107174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7921.691559] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7925.119532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7925.215921] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7925.218545] Lustre: Skipped 1 previous similar message [ 7925.282173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7926.946074] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7929.901772] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7932.218665] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7934.882758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7935.032456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7937.361021] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7940.348965] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7941.171292] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7947.248943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7949.282977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7949.290457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7949.290833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7949.298522] Lustre: Skipped 1 previous similar message [ 7955.363465] Lustre: server umount lustre-OST0000 complete [ 7959.520603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7959.521402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7959.527123] Lustre: Skipped 2 previous similar messages [ 7959.530367] Lustre: Skipped 2 previous similar messages [ 7963.356342] Lustre: server umount lustre-MDT0000 complete [ 7964.640914] LustreError: 406514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7964.652093] LustreError: 406514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7965.153834] LustreError: 406496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781612947 with bad export cookie 14896557052993716611 [ 7965.157115] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7965.161836] LustreError: 406496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7965.315710] Lustre: server umount lustre-MDT0001 complete [ 7969.972715] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7971.340380] Key type lgssc unregistered [ 7971.489628] LNet: 410237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7971.494476] LNetError: 410237:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7971.504906] LNet: Removed LNI 192.168.204.106@tcp [ 7971.878609] Key type .llcrypt unregistered [ 7971.881086] Key type ._llcrypt unregistered [ 7981.709804] Key type ._llcrypt registered [ 7981.711092] Key type .llcrypt registered [ 7981.757749] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7981.876801] Key type .llcrypt unregistered [ 7981.878603] Key type ._llcrypt unregistered [ 7989.875728] Key type ._llcrypt registered [ 7989.877123] Key type .llcrypt registered [ 7989.929942] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7990.439360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7990.462984] alg: No test for adler32 (adler32-zlib) [ 7991.377081] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 7991.495747] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 7993.111249] Key type lgssc registered [ 7993.650872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7997.450552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7997.465786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7998.736887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8000.605946] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8004.483128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8004.642654] LustreError: 412423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8004.651436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8004.651985] LustreError: 412423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8007.366856] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8008.738557] LustreError: 412427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8008.746908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8008.749639] LustreError: 412427:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8010.611377] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8013.531799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8013.706888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8015.537326] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8024.035969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8024.036101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8024.040674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8024.041223] Lustre: Skipped 1 previous similar message [ 8029.154347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8029.160059] Lustre: Skipped 2 previous similar messages [ 8029.430600] Lustre: server umount lustre-OST0000 complete [ 8033.765260] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8033.770796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8033.779455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8037.493414] Lustre: server umount lustre-MDT0000 complete [ 8039.140583] LustreError: 412408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781613021 with bad export cookie 12066917623294933659 [ 8039.143235] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8039.147490] LustreError: 412408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8039.300529] Lustre: server umount lustre-MDT0001 complete [ 8043.883714] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8045.206520] Key type lgssc unregistered [ 8045.365929] LNet: 415571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8045.369759] LNetError: 415571:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8046.437858] LNet: Removed LNI 192.168.204.106@tcp [ 8046.839490] Key type .llcrypt unregistered [ 8046.842164] Key type ._llcrypt unregistered [ 8059.114399] Key type ._llcrypt registered [ 8059.116419] Key type .llcrypt registered [ 8059.169585] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8059.661441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8059.669766] alg: No test for adler32 (adler32-zlib) [ 8060.554443] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 8060.662502] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 8062.256528] Key type lgssc registered [ 8062.713768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8066.174595] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8066.185873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8081.695554] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8086.815189] LustreError: 416939:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.106@tcp: failed processing log, type 4: rc = -110 [ 8113.439219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8117.780391] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8123.945340] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8129.475402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8129.777184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8129.831757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8134.889960] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8135.141329] LustreError: 416965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8135.158575] LustreError: 416965:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8140.266460] LustreError: 416964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8143.395543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8143.871651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8149.072534] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8157.190461] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8164.157467] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8173.490329] Lustre: server umount lustre-OST0000 complete [ 8179.172647] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8179.178830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8179.189957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8179.681248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8179.690163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8184.801389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8186.911506] Lustre: server umount lustre-MDT0000 complete [ 8189.142371] LustreError: 419465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781613171 with bad export cookie 15478684184997918061 [ 8189.148084] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8189.151097] LustreError: 419465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8199.759097] Lustre: server umount lustre-MDT0001 complete [ 8205.765353] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8207.357692] Key type lgssc unregistered [ 8207.519871] LNet: 420507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8207.524359] LNetError: 420507:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8207.538555] LNet: Removed LNI 192.168.204.106@tcp [ 8208.004580] Key type .llcrypt unregistered [ 8208.007602] Key type ._llcrypt unregistered [ 8222.719821] Key type ._llcrypt registered [ 8222.722127] Key type .llcrypt registered [ 8222.783142] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8223.417840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8223.441446] alg: No test for adler32 (adler32-zlib) [ 8224.415866] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 8224.581630] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 8226.215327] Key type lgssc registered [ 8226.872228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8231.735261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8231.748487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8233.040100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8235.832209] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8241.236987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8241.487433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8244.460146] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8248.915342] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8252.398979] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8256.754546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8256.959790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8258.026154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8260.541565] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8264.687460] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8267.231840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8267.234907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8267.242975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8271.251904] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.6@tcp (stopping) [ 8271.331220] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8271.341224] Lustre: Skipped 1 previous similar message [ 8273.368433] Lustre: server umount lustre-MDT0000 complete [ 8274.807703] LustreError: 421880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.6@tcp arrived at 1781613256 with bad export cookie 16925593750160425865 [ 8274.820789] LustreError: 421880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8275.915482] LustreError: 423745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781613257 with bad export cookie 16925593750160424976 [ 8275.924369] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8275.925623] LustreError: 423745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8276.448372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8276.449065] LustreError: 422629:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8276.455819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8276.463310] LustreError: 422629:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8276.464947] Lustre: Skipped 3 previous similar messages [ 8280.480978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8280.481748] LustreError: 421901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8280.494208] LustreError: 421901:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8282.114785] Lustre: server umount lustre-OST0000 complete [ 8285.603459] LustreError: 421895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8285.617793] LustreError: 421895:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8286.050441] Lustre: server umount lustre-MDT0001 complete [ 8291.912114] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8293.655298] Key type lgssc unregistered [ 8293.867079] LNet: 425584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8293.874098] LNetError: 425584:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8293.892893] LNet: Removed LNI 192.168.204.106@tcp [ 8294.430622] Key type .llcrypt unregistered [ 8294.432436] Key type ._llcrypt unregistered [ 8310.514022] Key type ._llcrypt registered [ 8310.517760] Key type .llcrypt registered [ 8310.589424] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8311.351439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8311.385891] alg: No test for adler32 (adler32-zlib) [ 8312.350897] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 8312.514036] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 8314.151155] Key type lgssc registered [ 8314.910080] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8319.965517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8319.984482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8321.296464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8324.439416] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8330.544825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8330.811992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8333.532189] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8338.065698] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8341.273390] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8345.098202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8345.303544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8346.358609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8349.041207] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8353.401446] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8357.344388] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8357.354171] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8357.365616] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8360.856624] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.6@tcp (stopping) [ 8360.935794] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8363.229817] Lustre: server umount lustre-MDT0001 complete [ 8366.049766] LustreError: 426974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8366.065044] LustreError: 426974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8367.072662] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8367.082750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8367.094754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8367.099635] Lustre: Skipped 2 previous similar messages [ 8371.168458] LustreError: 426980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8371.168697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8373.036115] Lustre: server umount lustre-OST0000 complete [ 8376.105873] Lustre: server umount lustre-MDT0000 complete [ 8386.117344] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8388.273957] Key type lgssc unregistered [ 8388.564325] LNet: 430663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8388.572322] LNetError: 430663:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8388.590939] LNet: Removed LNI 192.168.204.106@tcp [ 8389.242242] Key type .llcrypt unregistered [ 8389.244969] Key type ._llcrypt unregistered [ 8406.805721] Key type ._llcrypt registered [ 8406.807288] Key type .llcrypt registered [ 8406.878641] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 8417.843132] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8418.504225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8418.583639] alg: No test for adler32 (adler32-zlib) [ 8419.585932] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 8419.770713] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 8421.455342] Key type lgssc registered [ 8422.484244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8449.738871] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8459.896664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8459.931586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8461.155197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8461.196163] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8461.267435] Lustre: lustre-MDT0000: new disk, initializing [ 8461.334086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8461.347441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8464.108117] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8474.799612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8475.000763] Lustre: lustre-OST0000: new disk, initializing [ 8475.003833] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8475.006913] Lustre: Skipped 1 previous similar message [ 8475.012124] Lustre: 435671:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8475.079947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8479.012583] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8484.913262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8484.921548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8484.950324] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8485.843022] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8496.655238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8496.746106] Lustre: 434655:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8496.804674] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8496.876539] Lustre: lustre-MDT0001: new disk, initializing [ 8496.960155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8497.003945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 8497.010302] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 8500.192531] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8507.033259] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8507.171963] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8510.424219] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8510.589795] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8514.454515] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.6@tcp (stopping) [ 8518.117234] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8518.121234] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8518.124043] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8518.124837] Lustre: Skipped 1 previous similar message [ 8519.554410] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.6@tcp (stopping) [ 8519.559386] Lustre: Skipped 1 previous similar message [ 8523.234579] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8523.238226] Lustre: Skipped 1 previous similar message [ 8528.351281] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8528.354868] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8528.372946] Lustre: Skipped 2 previous similar messages [ 8528.472454] Lustre: server umount lustre-MDT0001 complete [ 8538.594145] LustreError: 434663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8538.613637] LustreError: 434663:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8543.713955] LustreError: 434667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8543.715084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8543.731957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8548.833979] LustreError: 434666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8549.906200] Lustre: server umount lustre-OST0000 complete [ 8559.123552] Lustre: server umount lustre-MDT0000 complete [ 8568.833166] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8570.973622] Key type lgssc unregistered [ 8571.212590] LNet: 439422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8571.220485] LNetError: 439422:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8571.255143] LNet: Removed LNI 192.168.204.106@tcp [ 8571.963429] Key type .llcrypt unregistered [ 8571.968714] Key type ._llcrypt unregistered [ 8599.201200] Key type ._llcrypt registered [ 8599.202969] Key type .llcrypt registered [ 8599.305480] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8599.577431] Key type .llcrypt unregistered [ 8599.580251] Key type ._llcrypt unregistered [ 8612.285931] Key type ._llcrypt registered [ 8612.288129] Key type .llcrypt registered [ 8612.357087] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8613.185302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8613.210528] alg: No test for adler32 (adler32-zlib) [ 8614.221883] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 8614.389096] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 8616.040569] Key type lgssc registered [ 8616.982877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8638.715868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8638.747584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8639.938564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8639.979300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8640.051199] Lustre: lustre-MDT0000: new disk, initializing [ 8640.113632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8640.132257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8643.343821] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8651.718341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8651.810533] Lustre: 442777:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8651.859104] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8651.862306] Lustre: Skipped 1 previous similar message [ 8651.911534] Lustre: lustre-MDT0001: new disk, initializing [ 8651.993875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8652.037928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8652.044482] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8655.892949] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8664.095588] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8668.465468] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8674.090129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8674.290758] Lustre: lustre-OST0000: new disk, initializing [ 8674.294423] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8674.300481] Lustre: 444742:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8674.379198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8678.844841] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8681.515771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8681.526094] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8681.639726] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8686.231669] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8692.193794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8692.195393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8692.201182] Lustre: Skipped 1 previous similar message [ 8692.207075] Lustre: Skipped 2 previous similar messages [ 8697.354884] Lustre: server umount lustre-MDT0000 complete [ 8700.261583] LustreError: 442770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781613682 with bad export cookie 7022453118667406404 [ 8700.264628] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8700.271913] LustreError: 442770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8700.525768] Lustre: server umount lustre-MDT0001 complete [ 8713.705445] Lustre: server umount lustre-OST0000 complete [ 8721.640233] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 08:41:41 (1781613701) [ 8726.783243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8727.077784] LustreError: 446768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8727.093278] LustreError: 446768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8727.139963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8730.602352] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8732.129875] LustreError: 446769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8736.234722] LustreError: 446768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8737.200726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8737.461232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8740.640369] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8746.046042] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8750.546277] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8769.062881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8769.209915] Lustre: Found index 0 for lustre-OST0000, updating log [ 8769.368988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8771.443697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 8773.686973] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8779.118330] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8783.591352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8783.611381] Lustre: Skipped 1 previous similar message [ 8783.616571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8784.679111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8784.687307] Lustre: Skipped 1 previous similar message [ 8788.748657] Lustre: server umount lustre-OST0000 complete [ 8804.325984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8804.332136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8804.342522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8806.372560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8809.279412] Lustre: server umount lustre-MDT0000 complete [ 8810.466191] LustreError: 448584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8810.475358] LustreError: 448584:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8811.906030] LustreError: 446749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781613793 with bad export cookie 7022453118667407790 [ 8811.908967] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8811.918181] LustreError: 446749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8812.117284] Lustre: server umount lustre-MDT0001 complete [ 8817.900666] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 08:43:17 (1781613797) [ 8826.400126] LNet: 1 local NIs in recovery (showing 1): 192.168.204.106@tcp [ 9411.811511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9412.039559] LustreError: 453174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9412.054720] LustreError: 453174:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9412.074841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9413.936817] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9417.938051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9418.115855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9420.032247] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9423.397563] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9425.760863] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9442.028353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9442.165167] Lustre: Found index 0 for lustre-OST0000, updating log [ 9442.300658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9445.365283] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9448.919923] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9454.562576] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9454.568882] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9454.578482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9454.581948] Lustre: Skipped 2 previous similar messages [ 9456.040561] Lustre: server umount lustre-OST0000 complete [ 9458.785707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9458.787598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9458.794992] Lustre: Skipped 1 previous similar message [ 9458.800684] Lustre: Skipped 1 previous similar message [ 9459.809482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9459.815110] Lustre: Skipped 1 previous similar message [ 9461.856738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9461.860961] Lustre: Skipped 3 previous similar messages [ 9463.907230] LustreError: 453171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9463.923078] LustreError: 453171:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9463.970711] Lustre: server umount lustre-MDT0000 complete [ 9465.916578] LustreError: 453155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781614447 with bad export cookie 7022453118667408861 [ 9465.919342] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9465.925538] LustreError: 453155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9466.102398] Lustre: server umount lustre-MDT0001 complete [ 9466.399087] LNet: 1 local NIs in recovery (showing 1): 192.168.204.106@tcp [ 9485.921815] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 9490.278474] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 9506.283372] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 08:54:46 (1781614486) [ 9513.817781] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 9515.113815] Key type lgssc unregistered [ 9515.272578] LNet: 459776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9515.277315] LNetError: 459776:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9573.668670] LNet: Removed LNI 192.168.204.106@tcp [ 9574.080092] Key type .llcrypt unregistered [ 9574.081696] Key type ._llcrypt unregistered [ 9582.374638] Key type ._llcrypt registered [ 9582.376288] Key type .llcrypt registered [ 9582.436901] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 9582.908640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9582.923994] alg: No test for adler32 (adler32-zlib) [ 9583.817334] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 9583.927449] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 9585.527204] Key type lgssc registered [ 9586.000217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9587.884541] LNet: Added LNI 192.168.204.106@tcp100 [8/256/0/180] [ 9588.269702] LNet: Added LNI 192.168.204.106@tcp101 [8/256/0/180] [ 9588.715422] LNet: Added LNI 192.168.204.106@tcp102 [8/256/0/180] [ 9589.145879] LNet: Added LNI 192.168.204.106@tcp103 [8/256/0/180] [ 9589.578196] LNet: Added LNI 192.168.204.106@tcp104 [8/256/0/180] [ 9590.011093] LNet: Added LNI 192.168.204.106@tcp105 [8/256/0/180] [ 9590.439399] LNet: Added LNI 192.168.204.106@tcp106 [8/256/0/180] [ 9590.879687] LNet: Added LNI 192.168.204.106@tcp107 [8/256/0/180] [ 9591.336691] LNet: Added LNI 192.168.204.106@tcp108 [8/256/0/180] [ 9591.818825] LNet: Added LNI 192.168.204.106@tcp109 [8/256/0/180] [ 9592.309577] LNet: Added LNI 192.168.204.106@tcp110 [8/256/0/180] [ 9592.747583] LNet: Added LNI 192.168.204.106@tcp111 [8/256/0/180] [ 9593.211799] LNet: Added LNI 192.168.204.106@tcp112 [8/256/0/180] [ 9593.650655] LNet: Added LNI 192.168.204.106@tcp113 [8/256/0/180] [ 9594.074222] LNet: Added LNI 192.168.204.106@tcp114 [8/256/0/180] [ 9594.531884] LNet: Added LNI 192.168.204.106@tcp115 [8/256/0/180] [ 9594.974273] LNet: Added LNI 192.168.204.106@tcp116 [8/256/0/180] [ 9595.419294] LNet: Added LNI 192.168.204.106@tcp117 [8/256/0/180] [ 9595.843755] LNet: Added LNI 192.168.204.106@tcp118 [8/256/0/180] [ 9596.265459] LNet: Added LNI 192.168.204.106@tcp119 [8/256/0/180] [ 9596.717580] LNet: Added LNI 192.168.204.106@tcp120 [8/256/0/180] [ 9597.194198] LNet: Added LNI 192.168.204.106@tcp121 [8/256/0/180] [ 9597.674668] LNet: Added LNI 192.168.204.106@tcp122 [8/256/0/180] [ 9598.157357] LNet: Added LNI 192.168.204.106@tcp123 [8/256/0/180] [ 9598.619610] LNet: Added LNI 192.168.204.106@tcp124 [8/256/0/180] [ 9599.099796] LNet: Added LNI 192.168.204.106@tcp125 [8/256/0/180] [ 9599.532395] LNet: Added LNI 192.168.204.106@tcp126 [8/256/0/180] [ 9599.994492] LNet: Added LNI 192.168.204.106@tcp127 [8/256/0/180] [ 9600.406370] LNet: Added LNI 192.168.204.106@tcp128 [8/256/0/180] [ 9600.847874] LNet: Added LNI 192.168.204.106@tcp129 [8/256/0/180] [ 9601.262541] LNet: Added LNI 192.168.204.106@tcp130 [8/256/0/180] [ 9601.746105] LNet: Added LNI 192.168.204.106@tcp131 [8/256/0/180] [ 9602.196225] LNet: Added LNI 192.168.204.106@tcp132 [8/256/0/180] [ 9602.630071] LNet: Added LNI 192.168.204.106@tcp133 [8/256/0/180] [ 9603.104042] LNet: Added LNI 192.168.204.106@tcp134 [8/256/0/180] [ 9603.519726] LNet: Added LNI 192.168.204.106@tcp135 [8/256/0/180] [ 9606.669673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9606.688026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9607.760792] Lustre: 462774:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.106@tcp: can't add NID 192.168.204.106@tcp130: rc = -75 [ 9607.852578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9607.873243] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9607.913431] Lustre: lustre-MDT0000: new disk, initializing [ 9607.979272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9607.992688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9609.923348] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9615.439836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9615.544281] Lustre: 462792:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9615.576362] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9615.580367] Lustre: Skipped 1 previous similar message [ 9615.620790] Lustre: lustre-MDT0001: new disk, initializing [ 9615.636928] Lustre: 463590:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.106@tcp: can't add NID 192.168.204.106@tcp130: rc = -75 [ 9615.642322] Lustre: 463590:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 9615.701884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9615.720194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9615.728252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9617.740493] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9622.446188] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9623.561586] Lustre: 464291:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.106@tcp: can't add NID 192.168.204.106@tcp130: rc = -75 [ 9623.568243] Lustre: 464291:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 9624.917904] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9627.793583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9627.916038] Lustre: lustre-OST0000: new disk, initializing [ 9627.918698] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9627.924178] Lustre: 464762:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9627.970923] Lustre: 464767:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.106@tcp: can't add NID 192.168.204.106@tcp130: rc = -75 [ 9627.975278] Lustre: 464767:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 9627.997505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9630.464583] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9631.692764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9631.697467] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9631.747488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9634.921437] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9641.953746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9641.955274] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9641.956789] LustreError: Skipped 1 previous similar message [ 9641.962967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9641.966589] Lustre: Skipped 1 previous similar message [ 9647.076266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9647.079256] Lustre: Skipped 1 previous similar message [ 9652.191173] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9652.195095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9652.201499] Lustre: Skipped 1 previous similar message [ 9652.275372] Lustre: server umount lustre-OST0000 complete [ 9656.800353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9656.804574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9656.811945] Lustre: Skipped 1 previous similar message [ 9656.814936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9659.809334] Lustre: server umount lustre-MDT0000 complete [ 9661.379351] LustreError: 462784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781614643 with bad export cookie 10056759981965136414 [ 9661.381156] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9661.385795] LustreError: 462784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9661.531552] Lustre: server umount lustre-MDT0001 complete [ 9665.883561] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 9667.262417] Key type lgssc unregistered [ 9667.446561] LNet: 466675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9667.450908] LNetError: 466675:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9667.454710] LNet: Removed LNI 192.168.204.106@tcp [ 9668.511564] LNet: Removed LNI 192.168.204.106@tcp100 [ 9668.514975] LNetError: 466675:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9668.520492] LNetError: 466675:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9668.525229] LNet: Removed LNI 192.168.204.106@tcp101 [ 9669.535149] LNet: Removed LNI 192.168.204.106@tcp102 [ 9669.537581] LNetError: 466675:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9669.540691] LNetError: 466675:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9669.545080] LNet: Removed LNI 192.168.204.106@tcp103 [ 9670.559157] LNet: Removed LNI 192.168.204.106@tcp104 [ 9671.583145] LNet: Removed LNI 192.168.204.106@tcp105 [ 9671.586966] LNetError: 466675:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9671.591526] LNetError: 466675:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 9671.595652] LNet: Removed LNI 192.168.204.106@tcp106 [ 9672.607142] LNet: Removed LNI 192.168.204.106@tcp107 [ 9673.631135] LNet: Removed LNI 192.168.204.106@tcp108 [ 9674.655148] LNet: Removed LNI 192.168.204.106@tcp109 [ 9675.679142] LNet: Removed LNI 192.168.204.106@tcp110 [ 9675.682259] LNetError: 466675:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9675.687285] LNetError: 466675:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 9675.692344] LNet: Removed LNI 192.168.204.106@tcp111 [ 9676.703511] LNet: Removed LNI 192.168.204.106@tcp112 [ 9677.727156] LNet: Removed LNI 192.168.204.106@tcp113 [ 9677.729333] LNet: Removed LNI 192.168.204.106@tcp114 [ 9677.732105] LNet: Removed LNI 192.168.204.106@tcp115 [ 9677.735950] LNet: Removed LNI 192.168.204.106@tcp116 [ 9677.737749] LNet: Removed LNI 192.168.204.106@tcp117 [ 9677.739809] LNet: Removed LNI 192.168.204.106@tcp118 [ 9677.742769] LNet: Removed LNI 192.168.204.106@tcp119 [ 9677.745171] LNet: Removed LNI 192.168.204.106@tcp120 [ 9677.747534] LNet: Removed LNI 192.168.204.106@tcp121 [ 9677.749554] LNet: Removed LNI 192.168.204.106@tcp122 [ 9677.751320] LNet: Removed LNI 192.168.204.106@tcp123 [ 9677.752839] LNet: Removed LNI 192.168.204.106@tcp124 [ 9677.754188] LNet: Removed LNI 192.168.204.106@tcp125 [ 9677.756253] LNet: Removed LNI 192.168.204.106@tcp126 [ 9677.758114] LNet: Removed LNI 192.168.204.106@tcp127 [ 9677.760253] LNet: Removed LNI 192.168.204.106@tcp128 [ 9677.762207] LNet: Removed LNI 192.168.204.106@tcp129 [ 9677.764106] LNet: Removed LNI 192.168.204.106@tcp130 [ 9677.766194] LNet: Removed LNI 192.168.204.106@tcp131 [ 9677.767536] LNet: Removed LNI 192.168.204.106@tcp132 [ 9677.769331] LNet: Removed LNI 192.168.204.106@tcp133 [ 9677.771230] LNet: Removed LNI 192.168.204.106@tcp134 [ 9677.780859] LNet: Removed LNI 192.168.204.106@tcp135 [ 9678.128588] Key type .llcrypt unregistered [ 9678.130329] Key type ._llcrypt unregistered [ 9689.034345] Key type ._llcrypt registered [ 9689.035813] Key type .llcrypt registered [ 9689.082812] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 9696.649514] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 9697.129175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9697.137646] alg: No test for adler32 (adler32-zlib) [ 9698.028437] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 9698.142966] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 9699.751297] Key type lgssc registered [ 9700.352972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9714.800935] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 08:58:15 (1781614695) [ 9721.746333] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 9722.969519] Key type lgssc unregistered [ 9723.110463] LNet: 470731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9723.113677] LNetError: 470731:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9723.123259] LNet: Removed LNI 192.168.204.106@tcp [ 9723.428516] Key type .llcrypt unregistered [ 9723.430505] Key type ._llcrypt unregistered [ 9730.899602] Key type ._llcrypt registered [ 9730.902551] Key type .llcrypt registered [ 9730.948073] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 9731.375299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9731.438548] alg: No test for adler32 (adler32-zlib) [ 9732.333398] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 9732.434523] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 9734.039177] Key type lgssc registered [ 9734.523860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9738.391555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9738.399943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9739.511211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9739.526584] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9739.567086] Lustre: lustre-MDT0000: new disk, initializing [ 9739.612147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9739.623065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9741.296308] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9746.105813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9746.160551] Lustre: 471920:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9746.193713] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9746.197014] Lustre: Skipped 1 previous similar message [ 9746.234919] Lustre: lustre-MDT0001: new disk, initializing [ 9746.275290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9746.297940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9746.302730] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9748.006647] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9752.369652] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9754.641524] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9757.590066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9757.715747] Lustre: lustre-OST0000: new disk, initializing [ 9757.719516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9757.722815] Lustre: 473890:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9757.760547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9760.174327] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9762.826291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9762.831375] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9762.874956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9764.764672] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9766.287823] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing lnet_if_list [ 9768.404931] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 9768.420224] LNet: Added LNI 192.168.204.106@tcp10 [8/256/0/180] [ 9770.710313] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 9770.726541] LNet: Added LNI 192.168.204.106@tcp11 [8/256/0/180] [ 9772.498994] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 9772.515238] LNet: Added LNI 192.168.204.106@tcp12 [8/256/0/180] [ 9774.258479] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 9774.273282] LNet: Added LNI 192.168.204.106@tcp13 [8/256/0/180] [ 9776.078886] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 9776.094729] LNet: Added LNI 192.168.204.106@tcp14 [8/256/0/180] [ 9778.025876] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 9778.040299] LNet: Added LNI 192.168.204.106@tcp15 [8/256/0/180] [ 9779.840236] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 9779.855268] LNet: Added LNI 192.168.204.106@tcp16 [8/256/0/180] [ 9781.737976] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 9781.751777] LNet: Added LNI 192.168.204.106@tcp17 [8/256/0/180] [ 9783.698334] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 9783.715240] LNet: Added LNI 192.168.204.106@tcp18 [8/256/0/180] [ 9785.692444] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 9785.708408] LNet: Added LNI 192.168.204.106@tcp19 [8/256/0/180] [ 9787.780033] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 9787.795160] LNet: Added LNI 192.168.204.106@tcp20 [8/256/0/180] [ 9789.812749] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 9789.826812] LNet: Added LNI 192.168.204.106@tcp21 [8/256/0/180] [ 9791.955784] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 9791.967092] LNet: Added LNI 192.168.204.106@tcp22 [8/256/0/180] [ 9794.145360] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 9794.159856] LNet: Added LNI 192.168.204.106@tcp23 [8/256/0/180] [ 9796.515511] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 9796.532544] LNet: Added LNI 192.168.204.106@tcp24 [8/256/0/180] [ 9798.786204] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 9798.805665] LNet: Added LNI 192.168.204.106@tcp25 [8/256/0/180] [ 9801.077209] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 9801.093559] LNet: Added LNI 192.168.204.106@tcp26 [8/256/0/180] [ 9803.419504] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 9803.432475] LNet: Added LNI 192.168.204.106@tcp27 [8/256/0/180] [ 9805.745069] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 9805.760451] LNet: Added LNI 192.168.204.106@tcp28 [8/256/0/180] [ 9808.187843] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 9808.204630] LNet: Added LNI 192.168.204.106@tcp29 [8/256/0/180] [ 9810.686500] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 9810.702345] LNet: Added LNI 192.168.204.106@tcp30 [8/256/0/180] [ 9813.104249] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 9813.119712] LNet: Added LNI 192.168.204.106@tcp31 [8/256/0/180] [ 9815.525530] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 9815.542387] LNet: Added LNI 192.168.204.106@tcp32 [8/256/0/180] [ 9818.172854] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 9818.186155] LNet: Added LNI 192.168.204.106@tcp33 [8/256/0/180] [ 9820.783627] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 9820.798602] LNet: Added LNI 192.168.204.106@tcp34 [8/256/0/180] [ 9823.488487] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 9823.503309] LNet: Added LNI 192.168.204.106@tcp35 [8/256/0/180] [ 9826.178460] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 9826.192910] LNet: Added LNI 192.168.204.106@tcp36 [8/256/0/180] [ 9828.970278] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 9828.991102] LNet: Added LNI 192.168.204.106@tcp37 [8/256/0/180] [ 9831.714611] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 9831.731379] LNet: Added LNI 192.168.204.106@tcp38 [8/256/0/180] [ 9834.589647] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 9834.604464] LNet: Added LNI 192.168.204.106@tcp39 [8/256/0/180] [ 9837.533132] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 9837.549455] LNet: Added LNI 192.168.204.106@tcp40 [8/256/0/180] [ 9840.536528] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 9840.551300] LNet: Added LNI 192.168.204.106@tcp41 [8/256/0/180] [ 9843.592907] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 9843.608628] LNet: Added LNI 192.168.204.106@tcp42 [8/256/0/180] [ 9846.522761] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 9846.539267] LNet: Added LNI 192.168.204.106@tcp43 [8/256/0/180] [ 9849.522400] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 9849.539753] LNet: Added LNI 192.168.204.106@tcp44 [8/256/0/180] [ 9852.710750] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 9852.724342] LNet: Added LNI 192.168.204.106@tcp45 [8/256/0/180] [ 9855.822966] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 9855.837440] LNet: Added LNI 192.168.204.106@tcp46 [8/256/0/180] [ 9858.982890] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 9858.999018] LNet: Added LNI 192.168.204.106@tcp47 [8/256/0/180] [ 9862.016156] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 9862.031881] LNet: Added LNI 192.168.204.106@tcp48 [8/256/0/180] [ 9865.001022] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 9865.015180] LNet: Added LNI 192.168.204.106@tcp49 [8/256/0/180] [ 9868.187661] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 9868.201387] LNet: Added LNI 192.168.204.106@tcp50 [8/256/0/180] [ 9879.520058] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9879.524254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9879.531269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9880.545444] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9880.547605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9880.554387] Lustre: Skipped 1 previous similar message [ 9880.560897] Lustre: Skipped 2 previous similar messages [ 9882.597484] Lustre: server umount lustre-MDT0000 complete [ 9884.118087] LustreError: 472719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781614865 with bad export cookie 7133408292751593355 [ 9884.122299] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9884.123981] LustreError: 472719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9884.259141] Lustre: server umount lustre-MDT0001 complete [ 9892.380288] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 9895.949972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9902.047381] Lustre: 471194:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781614867/real 1781614867] req@00000000a670be54 x1868158430438400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781614883 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9902.061041] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9906.079153] Lustre: 471195:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781614871/real 1781614871] req@0000000004d90c17 x1868158430438528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781614887 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9909.216132] LustreError: 471193:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006de174d0 x1868158430439552/t0(0) o250->MGC192.168.204.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9909.356127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9910.368108] Lustre: 471196:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781614876/real 1781614876] req@0000000070ba38be x1868158430438912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781614892 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9910.378707] Lustre: 471196:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9910.865069] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9914.132357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9914.295530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9915.872119] Lustre: 471194:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781614881/real 1781614881] req@000000009d04c4d8 x1868158430439040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781614897 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9915.909869] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9918.727924] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9919.463347] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9920.489627] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9920.780103] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9929.696756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9929.704412] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9929.711452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9935.202077] Lustre: server umount lustre-OST0000 complete [ 9939.936133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9939.939424] LustreError: Skipped 1 previous similar message [ 9939.941539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9939.946733] Lustre: Skipped 1 previous similar message [ 9939.949215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9939.951675] Lustre: Skipped 2 previous similar messages [ 9942.874394] Lustre: server umount lustre-MDT0000 complete [ 9944.352478] LustreError: 472719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781614926 with bad export cookie 7133408292751595525 [ 9944.354647] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9944.361309] LustreError: 472719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9944.502932] Lustre: server umount lustre-MDT0001 complete [ 9948.702320] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 9949.910754] Key type lgssc unregistered [ 9950.099595] LNet: 500865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9950.103469] LNetError: 500865:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9950.107239] LNet: Removed LNI 192.168.204.106@tcp [ 9950.109381] LNet: Removed LNI 192.168.204.106@tcp10 [ 9950.111349] LNet: Removed LNI 192.168.204.106@tcp11 [ 9950.113327] LNet: Removed LNI 192.168.204.106@tcp12 [ 9950.115224] LNet: Removed LNI 192.168.204.106@tcp13 [ 9950.116958] LNet: Removed LNI 192.168.204.106@tcp14 [ 9950.118612] LNet: Removed LNI 192.168.204.106@tcp15 [ 9950.120303] LNet: Removed LNI 192.168.204.106@tcp16 [ 9950.122386] LNet: Removed LNI 192.168.204.106@tcp17 [ 9950.124219] LNet: Removed LNI 192.168.204.106@tcp18 [ 9950.126218] LNet: Removed LNI 192.168.204.106@tcp19 [ 9950.127834] LNet: Removed LNI 192.168.204.106@tcp20 [ 9950.129252] LNet: Removed LNI 192.168.204.106@tcp21 [ 9950.130722] LNet: Removed LNI 192.168.204.106@tcp22 [ 9950.132081] LNet: Removed LNI 192.168.204.106@tcp23 [ 9950.134053] LNet: Removed LNI 192.168.204.106@tcp24 [ 9950.135835] LNet: Removed LNI 192.168.204.106@tcp25 [ 9950.137446] LNet: Removed LNI 192.168.204.106@tcp26 [ 9950.139106] LNet: Removed LNI 192.168.204.106@tcp27 [ 9950.140720] LNet: Removed LNI 192.168.204.106@tcp28 [ 9950.142244] LNet: Removed LNI 192.168.204.106@tcp29 [ 9950.143706] LNet: Removed LNI 192.168.204.106@tcp30 [ 9950.145464] LNet: Removed LNI 192.168.204.106@tcp31 [ 9950.147252] LNet: Removed LNI 192.168.204.106@tcp32 [ 9950.148883] LNet: Removed LNI 192.168.204.106@tcp33 [ 9950.150300] LNet: Removed LNI 192.168.204.106@tcp34 [ 9950.151842] LNet: Removed LNI 192.168.204.106@tcp35 [ 9950.153297] LNet: Removed LNI 192.168.204.106@tcp36 [ 9950.155200] LNet: Removed LNI 192.168.204.106@tcp37 [ 9950.156862] LNet: Removed LNI 192.168.204.106@tcp38 [ 9950.158367] LNet: Removed LNI 192.168.204.106@tcp39 [ 9950.159897] LNet: Removed LNI 192.168.204.106@tcp40 [ 9950.161594] LNet: Removed LNI 192.168.204.106@tcp41 [ 9950.163284] LNet: Removed LNI 192.168.204.106@tcp42 [ 9950.165104] LNet: Removed LNI 192.168.204.106@tcp43 [ 9950.166905] LNet: Removed LNI 192.168.204.106@tcp44 [ 9950.168943] LNet: Removed LNI 192.168.204.106@tcp45 [ 9950.170435] LNet: Removed LNI 192.168.204.106@tcp46 [ 9950.172155] LNet: Removed LNI 192.168.204.106@tcp47 [ 9950.173774] LNet: Removed LNI 192.168.204.106@tcp48 [ 9950.175206] LNet: Removed LNI 192.168.204.106@tcp49 [ 9950.179320] LNet: Removed LNI 192.168.204.106@tcp50 [ 9950.477121] Key type .llcrypt unregistered [ 9950.478395] Key type ._llcrypt unregistered [ 9958.342571] Key type ._llcrypt registered [ 9958.344458] Key type .llcrypt registered [ 9958.388700] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 9965.205942] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 9965.632982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9965.658598] alg: No test for adler32 (adler32-zlib) [ 9966.524898] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [ 9966.617907] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 9968.207279] Key type lgssc registered [ 9968.673157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9982.355712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9982.363961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9983.464528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9983.479332] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9983.521626] Lustre: lustre-MDT0000: new disk, initializing [ 9983.557553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9983.567637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9985.162866] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9990.166867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9990.218080] Lustre: 504035:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9990.246371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9990.248642] Lustre: Skipped 1 previous similar message [ 9990.287968] Lustre: lustre-MDT0001: new disk, initializing [ 9990.344829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9990.361625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9990.366162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9991.986206] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9996.123297] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9998.244651] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10001.350688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10001.436130] Lustre: *** cfs_fail_loc=90f, val=0*** [10001.471183] Lustre: lustre-OST0000: new disk, initializing [10001.473714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10001.477782] Lustre: 506050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10001.518544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10003.835799] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10008.073559] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10008.579542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10008.583918] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10008.629197] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10013.665500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10013.671227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10013.680879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10013.686095] Lustre: Skipped 1 previous similar message [10016.991102] Lustre: server umount lustre-OST0000 complete [10018.785096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10018.785764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10018.790502] Lustre: Skipped 2 previous similar messages [10023.904505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10023.907343] Lustre: Skipped 2 previous similar messages [10024.543193] Lustre: server umount lustre-MDT0000 complete [10025.997534] LustreError: 504029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781615007 with bad export cookie 1452328964306878430 [10025.999533] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10026.003922] LustreError: 504029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10026.159755] Lustre: server umount lustre-MDT0001 complete [10030.460877] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [10031.567118] Key type lgssc unregistered [10031.716487] LNet: 507951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10031.719784] LNetError: 507951:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10031.728286] LNet: Removed LNI 192.168.204.106@tcp [10032.057103] Key type .llcrypt unregistered [10032.058546] Key type ._llcrypt unregistered [10040.599305] Key type ._llcrypt registered [10040.600764] Key type .llcrypt registered [10040.647875] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [10040.986267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10041.040671] alg: No test for adler32 (adler32-zlib) [10041.920275] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [10042.039395] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [10043.647221] Key type lgssc registered [10044.131509] Lustre: Echo OBD driver; http://www.lustre.org/ [10047.097189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10047.104887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10048.315669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10049.896757] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10053.162934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10053.334604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10054.943236] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10058.008981] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10060.243979] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10062.860711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10063.014503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10065.404225] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10068.204886] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10077.359630] loop0: detected capacity change from 0 to 204800000 [10077.431086] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10077.436347] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10077.442501] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10079.836953] loop0: detected capacity change from 0 to 204800000 [10079.963468] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10079.969707] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10079.975964] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10082.046364] loop0: detected capacity change from 0 to 204800000 [10084.183603] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10084.242259] Lustre: fs15246-OST0000: new disk, initializing [10084.244692] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [10084.248032] Lustre: 512791:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10084.273807] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10086.654079] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10090.991654] loop1: detected capacity change from 0 to 204800000 [10093.077855] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10093.122039] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [10093.150604] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [10093.190728] Lustre: fs15246-MDT0000: new disk, initializing [10093.254443] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10093.266882] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10093.309867] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10094.367995] Lustre: 513911:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000098270c4b x1868158754388608/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781615092 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [10094.825888] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10103.228606] LustreError: 513911:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [10103.234156] LustreError: 513911:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [10103.238396] LustreError: 513911:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [10106.851846] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10106.858264] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [10109.408681] Lustre: server umount fs15246-MDT0000 complete [10111.974943] LustreError: 509197:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10111.982118] LustreError: 509197:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10112.953498] Lustre: server umount fs15246-OST0000 complete [10117.090109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10117.096238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10117.097905] Lustre: Skipped 1 previous similar message [10122.208978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10122.214596] Lustre: Skipped 1 previous similar message [10123.024918] Lustre: server umount lustre-OST0000 complete [10125.280414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10125.283791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10125.290402] Lustre: Skipped 1 previous similar message [10125.292884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10127.328905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10130.529771] Lustre: server umount lustre-MDT0000 complete [10132.009971] LustreError: 509180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781615113 with bad export cookie 3625778927259465044 [10132.013610] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10132.015681] LustreError: 509180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10132.142622] Lustre: server umount lustre-MDT0001 complete [10136.247129] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [10137.388262] Key type lgssc unregistered [10137.515430] LNet: 517004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10137.518954] LNetError: 517004:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10137.528439] LNet: Removed LNI 192.168.204.106@tcp [10137.845743] Key type .llcrypt unregistered [10137.847325] Key type ._llcrypt unregistered [10146.214151] Key type ._llcrypt registered [10146.215668] Key type .llcrypt registered [10146.271117] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [10146.758251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10146.809539] alg: No test for adler32 (adler32-zlib) [10147.685482] Lustre: Lustre: Build Version: 2.17.54_40_g537638b [10147.782630] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [10149.375208] Key type lgssc registered [10149.872874] Lustre: Echo OBD driver; http://www.lustre.org/ [10165.021875] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [10168.646560] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [10182.052293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10182.059936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10183.162698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10183.177311] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10183.216515] Lustre: lustre-MDT0000: new disk, initializing [10183.248198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10183.257442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10184.831395] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10189.371889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10189.416104] Lustre: 521441:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10189.442241] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10189.444545] Lustre: Skipped 1 previous similar message [10189.481210] Lustre: lustre-MDT0001: new disk, initializing [10189.514221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10189.526431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10189.530351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10191.078293] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10195.053602] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10197.092868] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10199.543450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10199.652642] Lustre: lustre-OST0000: new disk, initializing [10199.654790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10199.657689] Lustre: 523407:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10199.689721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10201.844295] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10205.809511] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10207.379544] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10208.760832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10208.764913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10208.803642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10209.500952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [10211.102712] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10211.193589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10213.860175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10213.860552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10213.863047] LustreError: Skipped 1 previous similar message [10213.868669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10213.872514] Lustre: Skipped 1 previous similar message [10218.016364] Lustre: server umount lustre-OST0000 complete [10220.512334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10220.516183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10220.521467] Lustre: Skipped 1 previous similar message [10220.524134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10224.096710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10224.097582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10225.501122] Lustre: server umount lustre-MDT0000 complete [10226.917694] LustreError: 521432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781615208 with bad export cookie 9012033169885433761 [10226.921355] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10226.922686] LustreError: 521432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10227.043077] Lustre: server umount lustre-MDT0001 complete [10236.198332] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [10240.136565] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [10252.182509] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10000 sec ========= 09:07:12 (1781615232) [10252.816099] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:07:13 (1781615233) === [10254.130540] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:07:14 (1781615234) === [10268.888864] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [10270.001615] Key type lgssc unregistered [10270.140554] LNet: 530053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10270.143702] LNetError: 530053:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10270.151344] LNet: Removed LNI 192.168.204.106@tcp [10270.495822] Key type .llcrypt unregistered [10270.497227] Key type ._llcrypt unregistered