[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 494529371 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003170] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008714] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011013] pid_max: default: 32768 minimum: 301 [ 0.012133] LSM: Security Framework initializing [ 0.014009] Yama: becoming mindful. [ 0.015031] SELinux: Initializing. [ 0.016061] *** VALIDATE selinux *** [ 0.024752] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029041] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030140] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033063] *** VALIDATE tmpfs *** [ 0.034444] *** VALIDATE proc *** [ 0.035219] *** VALIDATE cgroup *** [ 0.036010] *** VALIDATE cgroup2 *** [ 0.037242] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038000] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039029] Spectre V2 : User space: Vulnerable [ 0.040020] Speculative Store Bypass: Vulnerable [ 0.043335] debug: unmapping init [mem 0xffffffff95059000-0xffffffff95060fff] [ 0.046172] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047703] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048021] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050012] ... generic registers: 4 [ 0.051012] ... value mask: 0000ffffffffffff [ 0.052014] ... max period: 00007fffffffffff [ 0.053014] ... fixed-purpose events: 3 [ 0.054012] ... event mask: 000000070000000f [ 0.055330] rcu: Hierarchical SRCU implementation. [ 0.057308] smp: Bringing up secondary CPUs ... [ 0.058476] x86: Booting SMP configuration: [ 0.059024] .... node #0, CPUs: #1 #2 #3 [ 0.063073] smp: Brought up 1 node, 4 CPUs [ 0.065014] smpboot: Max logical packages: 1 [ 0.066020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.129682] node 0 deferred pages initialised in 59ms [ 0.133167] devtmpfs: initialized [ 0.134242] x86/mm: Memory block size: 128MB [ 0.137753] gcov: version magic: 0x41383552 [ 0.140144] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.143083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146268] pinctrl core: initialized pinctrl subsystem [ 0.147136] [ 0.147750] ************************************************************* [ 0.150011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152013] ** ** [ 0.155013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157010] ** ** [ 0.159012] ** This means that this kernel is built to expose internal ** [ 0.161011] ** IOMMU data structures, which may compromise security on ** [ 0.163012] ** your system. ** [ 0.166013] ** ** [ 0.168011] ** If you see this message and you are not debugging the ** [ 0.171019] ** kernel, report this immediately to your vendor! ** [ 0.173011] ** ** [ 0.175012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178013] ************************************************************* [ 0.180700] NET: Registered protocol family 16 [ 0.182436] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.185041] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.187057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.191036] cpuidle: using governor menu [ 0.193303] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.194349] PCI: Using configuration type 1 for base access [ 0.196126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.204177] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.205037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.207073] cryptd: max_cpu_qlen set to 1000 [ 0.208188] ACPI: Added _OSI(Module Device) [ 0.210029] ACPI: Added _OSI(Processor Device) [ 0.212044] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.213013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.219516] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.227266] ACPI: Interpreter enabled [ 0.229090] ACPI: PM: (supports S0 S3 S4 S5) [ 0.231013] ACPI: Using IOAPIC for interrupt routing [ 0.232109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.235443] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.245666] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.248046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.251020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.254074] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.258265] acpiphp: Slot [2] registered [ 0.260114] acpiphp: Slot [5] registered [ 0.262128] acpiphp: Slot [6] registered [ 0.264164] acpiphp: Slot [7] registered [ 0.265109] acpiphp: Slot [8] registered [ 0.267110] acpiphp: Slot [9] registered [ 0.268106] acpiphp: Slot [10] registered [ 0.270121] acpiphp: Slot [3] registered [ 0.271082] acpiphp: Slot [4] registered [ 0.273126] acpiphp: Slot [11] registered [ 0.274094] acpiphp: Slot [12] registered [ 0.275092] acpiphp: Slot [13] registered [ 0.277092] acpiphp: Slot [14] registered [ 0.278091] acpiphp: Slot [15] registered [ 0.280098] acpiphp: Slot [16] registered [ 0.281105] acpiphp: Slot [17] registered [ 0.283140] acpiphp: Slot [18] registered [ 0.284085] acpiphp: Slot [19] registered [ 0.285135] acpiphp: Slot [20] registered [ 0.288118] acpiphp: Slot [21] registered [ 0.289079] acpiphp: Slot [22] registered [ 0.290075] acpiphp: Slot [23] registered [ 0.292112] acpiphp: Slot [24] registered [ 0.293083] acpiphp: Slot [25] registered [ 0.294075] acpiphp: Slot [26] registered [ 0.296097] acpiphp: Slot [27] registered [ 0.297099] acpiphp: Slot [28] registered [ 0.299123] acpiphp: Slot [29] registered [ 0.301129] acpiphp: Slot [30] registered [ 0.303198] acpiphp: Slot [31] registered [ 0.305091] PCI host bridge to bus 0000:00 [ 0.306024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.309038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.312032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.315039] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.318043] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.320035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.322308] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.325202] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.328787] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.340016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.345140] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.348028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.350028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.353028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.356644] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.359727] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.362049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.365846] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.371018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.384984] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.389025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.397087] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.411026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.419024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.438026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.450254] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.457016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.465017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.482034] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.492945] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.499021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.507014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.526014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.535599] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.541015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.547016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.562023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.570479] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.580015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.587015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.604016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.614621] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.621013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.628013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.644018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.658083] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.661455] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.663405] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.666361] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.669417] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.675134] iommu: Default domain type: Passthrough [ 0.677452] SCSI subsystem initialized [ 0.679177] ACPI: bus type USB registered [ 0.681096] usbcore: registered new interface driver usbfs [ 0.683222] usbcore: registered new interface driver hub [ 0.685080] usbcore: registered new device driver usb [ 0.687139] pps_core: LinuxPPS API ver. 1 registered [ 0.689010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.692083] PTP clock support registered [ 0.694173] EDAC MC: Ver: 3.0.0 [ 0.696154] PCI: Using ACPI for IRQ routing [ 0.699086] NetLabel: Initializing [ 0.701016] NetLabel: domain hash size = 128 [ 0.702018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.704133] NetLabel: unlabeled traffic allowed by default [ 0.707216] vgaarb: loaded [ 0.709300] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.711024] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.716340] clocksource: Switched to clocksource kvm-clock [ 0.828965] VFS: Disk quotas dquot_6.6.0 [ 0.830790] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.833473] *** VALIDATE ramfs *** [ 0.834813] *** VALIDATE hugetlbfs *** [ 0.837392] pnp: PnP ACPI init [ 0.840293] pnp: PnP ACPI: found 6 devices [ 0.856890] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.860980] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.863716] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.866374] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.869396] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.872489] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.876072] NET: Registered protocol family 2 [ 0.878465] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.884470] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.888239] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.893924] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.897917] TCP: Hash tables configured (established 65536 bind 65536) [ 0.901381] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.905150] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.907694] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.910803] NET: Registered protocol family 1 [ 0.914537] RPC: Registered named UNIX socket transport module. [ 0.916889] RPC: Registered udp transport module. [ 0.919272] RPC: Registered tcp transport module. [ 0.921575] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.924528] NET: Registered protocol family 44 [ 0.926446] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.928632] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.931095] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.933655] PCI: CLS 0 bytes, default 64 [ 0.935525] Unpacking initramfs... [ 2.419543] debug: unmapping init [mem 0xffff96263cc54000-0xffff96263ffbffff] [ 2.426182] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.428608] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.431854] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.940149] Initialise system trusted keyrings [ 2.942266] Key type blacklist registered [ 2.944323] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.953555] zbud: loaded [ 2.957170] *** VALIDATE nfs *** [ 2.958642] *** VALIDATE nfs4 *** [ 2.960504] pstore: using deflate compression [ 2.963972] Platform Keyring initialized [ 3.069080] NET: Registered protocol family 38 [ 3.070463] Key type asymmetric registered [ 3.071585] Asymmetric key parser 'x509' registered [ 3.072875] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.075501] io scheduler mq-deadline registered [ 3.077678] io scheduler kyber registered [ 3.079554] io scheduler bfq registered [ 3.082851] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.086472] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.089960] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.092988] ACPI: Power Button [PWRF] [ 3.098838] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.105827] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.121490] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.130495] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.150714] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.180053] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.208966] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.214309] Non-volatile memory driver v1.3 [ 3.216233] Linux agpgart interface v0.103 [ 3.247683] virtio_blk virtio1: [vda] 145880 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.251018] vda: detected capacity change from 0 to 74690560 [ 3.266382] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.269232] vdb: detected capacity change from 0 to 1073741824 [ 3.283148] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.286335] vdc: detected capacity change from 0 to 2621440000 [ 3.313170] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.316395] vdd: detected capacity change from 0 to 2621440000 [ 3.333354] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.336677] vde: detected capacity change from 0 to 4294967296 [ 3.359240] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.362719] vdf: detected capacity change from 0 to 4294967296 [ 3.370737] libphy: Fixed MDIO Bus: probed [ 3.376232] usbcore: registered new interface driver usbserial_generic [ 3.379351] usbserial: USB Serial support registered for generic [ 3.381803] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.386498] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.388614] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.391170] mousedev: PS/2 mouse device common for all mice [ 3.394232] rtc_cmos 00:05: RTC can wake from S4 [ 3.396492] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.401182] rtc_cmos 00:05: registered as rtc0 [ 3.403716] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.404123] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.408328] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.409325] intel_pstate: CPU model not supported [ 3.415343] hid: raw HID events driver (C) Jiri Kosina [ 3.417651] usbcore: registered new interface driver usbhid [ 3.420289] usbhid: USB HID core driver [ 3.422181] drop_monitor: Initializing network drop monitor service [ 3.424733] Initializing XFRM netlink socket [ 3.427347] NET: Registered protocol family 10 [ 3.430654] Segment Routing with IPv6 [ 3.432496] NET: Registered protocol family 17 [ 3.434506] mpls_gso: MPLS GSO support [ 3.439711] RAS: Correctable Errors collector initialized. [ 3.441536] AVX version of gcm_enc/dec engaged. [ 3.443578] AES CTR mode by8 optimization enabled [ 3.526534] sched_clock: Marking stable (3526515134, 0)->(4410718332, -884203198) [ 3.530197] registered taskstats version 1 [ 3.531652] Loading compiled-in X.509 certificates [ 3.533796] zswap: loaded using pool lzo/zbud [ 3.562799] Key type big_key registered [ 3.575212] Key type encrypted registered [ 3.577070] ima: No TPM chip found, activating TPM-bypass! [ 3.579099] ima: Allocated hash algorithm: sha1 [ 3.580700] ima: No architecture policies found [ 3.582522] evm: Initialising EVM extended attributes: [ 3.583899] evm: security.selinux [ 3.584887] evm: security.ima [ 3.586380] evm: security.capability [ 3.587727] evm: HMAC attrs: 0x1 [ 3.590482] rtc_cmos 00:05: setting system clock to 2026-09-10 23:24:14 UTC (1789082654) [ 3.596416] debug: unmapping init [mem 0xffffffff96003000-0xffffffff961fffff] [ 3.599634] debug: unmapping init [mem 0xffffffff94d82000-0xffffffff95058fff] [ 3.609090] Write protecting the kernel read-only data: 28672k [ 3.613550] debug: unmapping init [mem 0xffffffff93403000-0xffffffff935fffff] [ 3.616794] debug: unmapping init [mem 0xffffffff93d14000-0xffffffff93dfffff] [ 3.655407] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.665624] systemd[1]: Detected virtualization kvm. [ 3.667712] systemd[1]: Detected architecture x86-64. [ 3.669739] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.696888] systemd[1]: No hostname configured. [ 3.699084] systemd[1]: Set hostname to . [ 3.702782] random: systemd: uninitialized urandom read (16 bytes read) [ 3.704854] systemd[1]: Initializing machine ID from random generator. [ 3.825528] random: systemd: uninitialized urandom read (16 bytes read) [ 3.828754] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.833384] random: systemd: uninitialized urandom read (16 bytes read) [ 3.836827] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.841994] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.515147] device-mapper: uevent: version 1.0.3 [ 4.517651] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.211597] virtio_net virtio0 ens2: renamed from eth0 [ 5.227684] random: fast init done [ 5.269750] scsi host0: ata_piix [ 5.287146] scsi host1: ata_piix [ 5.288873] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.292564] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.584307] dracut-initqueue[595]: RTNETLINK answers: File exists [ 10.094287] random: crng init done [ 10.095487] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.453589] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.601619] printk: systemd: 22 output lines suppressed due to ratelimiting [ 11.844830] SELinux: Disabled at runtime. [ 11.905339] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.914635] systemd[1]: Detected virtualization kvm. [ 11.916494] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.413944] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.417261] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.421755] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.426321] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.430135] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.438301] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.450104] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ 12.508774] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.903031] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.208687] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.262848] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.389347] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.409368] EDAC sbridge: Ver: 1.1.2 [ 15.123470] Key type dns_resolver registered [ 15.437978] NFS: Registering the id_resolver key type [ 15.441135] Key type id_resolver registered [ 15.443671] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Login Service... Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg644-server login: [ 47.956318] libcfs: loading out-of-tree module taints kernel. [ 47.993746] Key type ._llcrypt registered [ 47.997418] Key type .llcrypt registered [ 48.125096] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 64.593206] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 65.932347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 65.969663] alg: No test for adler32 (adler32-zlib) [ 67.419839] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 68.589906] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 70.431199] Key type lgssc registered [ 72.684919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 76.424773] hrtimer: interrupt took 4766040 ns [ 93.749792] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 129.536283] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 141.986692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 142.040231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 143.407472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 143.451367] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 143.579179] Lustre: lustre-MDT0000: new disk, initializing [ 143.684249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 143.704405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 148.408514] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 162.208402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 162.339487] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 162.365859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 162.371599] Lustre: Skipped 1 previous similar message [ 162.435574] Lustre: lustre-MDT0001: new disk, initializing [ 162.507867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 162.542943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 162.558050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 166.986552] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.149321] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 181.679458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 181.924526] Lustre: lustre-OST0000: new disk, initializing [ 181.928718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 181.934888] Lustre: 8449:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 181.988287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.195378] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.996810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 191.003819] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 191.082086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 201.934605] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 202.065663] Lustre: lustre-OST0001: new disk, initializing [ 202.073234] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 202.077922] Lustre: 9523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 202.125831] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 208.258344] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.042491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 212.053299] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 212.115778] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 219.466625] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 226.160588] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 233.337289] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing check_logdir /tmp/testlogs/ [ 239.100994] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing yml_node [ 244.426523] Lustre: DEBUG MARKER: Client: 2.17.55.25 [ 246.987857] Lustre: DEBUG MARKER: MDS: 2.17.55.25 [ 249.629732] Lustre: DEBUG MARKER: OSS: 2.17.55.25 [ 251.268199] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 10 19:28:20 EDT 2026 [ 271.081149] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 272.902428] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 290.784118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 290.791196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.811777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 293.864784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 293.865134] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.881108] Lustre: Skipped 1 previous similar message [ 293.897488] Lustre: Skipped 1 previous similar message [ 295.849775] Lustre: server umount lustre-MDT0000 complete [ 300.311349] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789082951 with bad export cookie 9294659257717084828 [ 300.317707] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 300.336985] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 304.095902] LustreError: 12156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 304.096394] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.096909] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 304.096914] Lustre: Skipped 2 previous similar messages [ 304.124633] LustreError: 12156:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 304.169179] Lustre: Skipped 2 previous similar messages [ 305.121873] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 305.137686] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 306.806887] Lustre: server umount lustre-MDT0001 complete [ 310.780580] Lustre: server umount lustre-OST0000 complete [ 314.673791] Lustre: server umount lustre-OST0001 complete [ 321.886396] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 332.598910] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 370.275645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 370.509659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 370.529147] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 370.588304] Lustre: lustre-MDT0000: new disk, initializing [ 370.648910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 370.670483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 376.144044] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 388.125828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 388.283305] Lustre: 15078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 388.384973] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 388.388374] Lustre: Skipped 1 previous similar message [ 388.499526] Lustre: lustre-MDT0001: new disk, initializing [ 388.607485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 388.641373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 388.661319] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 392.618165] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 400.713414] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 405.771303] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 411.853810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 412.089867] Lustre: lustre-OST0000: new disk, initializing [ 412.098636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 412.107614] Lustre: 17046:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 412.233182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 414.000329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 414.016616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 414.193065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 417.394661] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 425.220788] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 429.194126] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 429.384643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 433.292786] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 433.507208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 439.782574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 439.783164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 439.789502] LustreError: Skipped 1 previous similar message [ 439.797197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 439.803684] Lustre: Skipped 4 previous similar messages [ 441.168616] Lustre: server umount lustre-OST0000 complete [ 444.897580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 444.903927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 444.904605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 444.912718] Lustre: Skipped 2 previous similar messages [ 450.664898] Lustre: server umount lustre-MDT0000 complete [ 454.348283] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789083105 with bad export cookie 9294659257717086361 [ 454.350643] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 454.354680] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 454.662786] Lustre: server umount lustre-MDT0001 complete [ 459.240841] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 19:31:48 (1789083108) [ 461.929666] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 463.726420] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 19:31:53 (1789083113) [ 470.568535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 471.065512] LustreError: 19454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 471.115608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.121509] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 476.131213] LustreError: 19455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 480.244627] LustreError: 19455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.886974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.823858] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 503.804125] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 512.189776] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 520.673699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 527.748025] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 535.837434] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 550.883730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 550.890968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 550.903477] Lustre: Skipped 1 previous similar message [ 550.925660] Lustre: Skipped 4 previous similar messages [ 556.599558] Lustre: server umount lustre-OST0000 complete [ 565.996279] Lustre: server umount lustre-MDT0000 complete [ 566.248626] LustreError: 19454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.272809] LustreError: 19454:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 569.606741] LustreError: 19437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789083220 with bad export cookie 9294659257717087243 [ 569.629464] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 569.942271] Lustre: server umount lustre-MDT0001 complete [ 581.653330] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 584.296561] Key type lgssc unregistered [ 584.526482] LNet: 23008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 584.535711] LNetError: 23008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 584.553332] LNet: Removed LNI 192.168.206.144@tcp [ 585.458150] Key type .llcrypt unregistered [ 585.466480] Key type ._llcrypt unregistered [ 608.047719] Key type ._llcrypt registered [ 608.050440] Key type .llcrypt registered [ 608.189941] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 609.509170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 609.548935] alg: No test for adler32 (adler32-zlib) [ 611.004664] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 611.342747] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 613.031177] Key type lgssc registered [ 615.000763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 625.070996] loop: module loaded [ 625.130727] loop0: detected capacity change from 0 to 204800000 [ 625.337422] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 625.346040] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 625.609983] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 631.921319] loop0: detected capacity change from 0 to 204800000 [ 632.110994] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 632.129535] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 632.148525] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 637.400153] loop0: detected capacity change from 0 to 204800000 [ 642.409729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 642.466494] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 643.654107] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 643.726313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 643.862811] Lustre: lustre-MDT0000: new disk, initializing [ 643.973213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 644.000773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 648.262932] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 656.183314] loop1: detected capacity change from 0 to 204800000 [ 661.047716] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 661.306742] Lustre: lustre-OST07c6: new disk, initializing [ 661.309310] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 661.312558] Lustre: Skipped 1 previous similar message [ 661.317685] Lustre: 26194:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 661.386320] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 666.800783] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 667.209197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 667.218559] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 667.260276] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 671.141258] Lustre: 25178:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 671.151655] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 671.159841] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 671.169585] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 671.181155] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 671.188724] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 691.241637] Lustre: 26820:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 691.255250] Lustre: 26820:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 691.266096] Lustre: 26820:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 691.282603] Lustre: 26820:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 691.300719] Lustre: 26820:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 691.310476] Lustre: 26820:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 696.522982] Lustre: 26820:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 696.545822] Lustre: 26820:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 696.569371] Lustre: 26820:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 696.588714] Lustre: 26820:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.607927] Lustre: 26820:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 696.618803] Lustre: 26820:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.623523] Lustre: 26820:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 696.631574] Lustre: 26820:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.641129] Lustre: 26820:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 696.649822] Lustre: 26820:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.669972] Lustre: 26820:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 696.683644] Lustre: 26820:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.846680] Lustre: 25179:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 699.862661] Lustre: 25179:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 699.869287] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 699.878602] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.889380] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 699.898688] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.904764] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 699.910799] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.916567] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 699.921300] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.926315] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 699.930374] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.860758] Lustre: 25179:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 704.874944] Lustre: 25179:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 704.884643] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 704.896833] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.907626] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 704.921958] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.924762] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 704.939349] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.956057] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 704.969532] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.977973] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 704.990635] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 713.479088] Lustre: 26820:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 713.493100] Lustre: 26820:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 713.501087] Lustre: 26820:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 713.517216] Lustre: 26820:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 713.527051] Lustre: 26820:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 713.540027] Lustre: 26820:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 713.545745] Lustre: 26820:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 713.553564] Lustre: 26820:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 713.562376] Lustre: 26820:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 713.567221] Lustre: 26820:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 713.574315] Lustre: 26820:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 713.582889] Lustre: 26820:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 769.514634] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 769.531589] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 769.555589] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 774.634990] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 779.233250] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 779.363244] Lustre: server umount lustre-OST07c6 complete [ 787.434515] Lustre: server umount lustre-MDT0000 complete [ 800.587937] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 802.972494] Key type lgssc unregistered [ 803.286362] LNet: 28751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.302108] LNetError: 28751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 803.322756] LNet: Removed LNI 192.168.206.144@tcp [ 804.137807] Key type .llcrypt unregistered [ 804.147727] Key type ._llcrypt unregistered [ 822.023838] Key type ._llcrypt registered [ 822.025903] Key type .llcrypt registered [ 822.111559] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 823.218898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 823.246772] alg: No test for adler32 (adler32-zlib) [ 824.371601] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 824.563917] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 826.215646] Key type lgssc registered [ 827.270976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 834.939948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 834.973838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 836.335942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 839.995636] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 846.821389] LustreError: 30020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 846.840055] LustreError: 30020:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 847.477417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 847.881213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 852.462873] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 859.167634] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 864.033820] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 869.872786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 870.184795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 875.797293] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 881.456253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 884.013332] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 902.112188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 902.119608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 902.133161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 902.147549] Lustre: Skipped 1 previous similar message [ 902.160336] Lustre: Skipped 1 previous similar message [ 904.328302] Lustre: server umount lustre-OST0000 complete [ 909.281529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 909.291190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 909.308721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 912.353857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 912.354262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 917.472661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 917.491317] Lustre: Skipped 2 previous similar messages [ 922.079156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 922.240210] Lustre: server umount lustre-MDT0000 complete [ 922.595836] LustreError: 30020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 922.620734] LustreError: 30020:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 925.471779] LustreError: 30002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789083576 with bad export cookie 1261491165738485867 [ 925.485238] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 925.725851] Lustre: server umount lustre-MDT0001 complete [ 934.466780] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 936.384909] Key type lgssc unregistered [ 936.593221] LNet: 33757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.599287] LNetError: 33757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.613583] LNet: Removed LNI 192.168.206.144@tcp [ 937.124739] Key type .llcrypt unregistered [ 937.127328] Key type ._llcrypt unregistered [ 950.490312] Key type ._llcrypt registered [ 950.492172] Key type .llcrypt registered [ 950.566106] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 951.344909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 951.378480] alg: No test for adler32 (adler32-zlib) [ 952.380300] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 952.565309] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 954.199165] Key type lgssc registered [ 955.117308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 961.153167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 961.174168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 962.504661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 965.803398] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 972.771100] LustreError: 35007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 972.797462] LustreError: 35007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 972.997854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 973.355978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 977.217777] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 983.220972] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 987.791370] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 993.600272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 993.877383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 999.892309] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1006.923859] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1007.222791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1017.824891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1017.833275] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1017.844674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1017.866773] Lustre: Skipped 1 previous similar message [ 1017.882332] Lustre: Skipped 1 previous similar message [ 1021.909327] Lustre: server umount lustre-OST0000 complete [ 1028.066148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1028.076538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1028.083707] Lustre: Skipped 1 previous similar message [ 1028.130196] Lustre: Skipped 1 previous similar message [ 1031.430980] Lustre: server umount lustre-MDT0000 complete [ 1033.189396] LustreError: 35007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1033.200388] LustreError: 35007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1034.951590] LustreError: 34989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789083685 with bad export cookie 783191174873244757 [ 1034.960902] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1034.965298] LustreError: 34989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1035.331164] Lustre: server umount lustre-MDT0001 complete [ 1044.614985] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 1046.938401] Key type lgssc unregistered [ 1047.194332] LNet: 38539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.201034] LNetError: 38539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.214717] LNet: Removed LNI 192.168.206.144@tcp [ 1047.969192] Key type .llcrypt unregistered [ 1047.971339] Key type ._llcrypt unregistered [ 1068.980426] Key type ._llcrypt registered [ 1068.982778] Key type .llcrypt registered [ 1069.090736] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 1069.967811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1070.028310] alg: No test for adler32 (adler32-zlib) [ 1071.076854] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 1071.278771] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 1072.912251] Key type lgssc registered [ 1073.851242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1080.777586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1080.788889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1082.238970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1086.279416] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1092.585490] LustreError: 39956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1092.597839] LustreError: 39956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1093.329133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1093.567923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1097.468227] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1104.117488] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1109.046202] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1114.883868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1115.173354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1120.177738] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1127.260604] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1127.474922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1134.573513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1134.587750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.599837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1137.638627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1137.639391] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.644356] Lustre: Skipped 1 previous similar message [ 1139.638647] Lustre: server umount lustre-MDT0000 complete [ 1139.985951] LustreError: 41388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1142.752877] LustreError: 41008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1142.778661] LustreError: 41008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1142.987654] LustreError: 39938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789083793 with bad export cookie 3677816610477520176 [ 1143.003052] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1145.098171] LustreError: 41388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1145.125626] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.44@tcp (stopping) [ 1145.129773] Lustre: Skipped 2 previous similar messages [ 1147.873183] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1147.894178] Lustre: Skipped 1 previous similar message [ 1147.903090] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1149.385796] Lustre: server umount lustre-MDT0001 complete [ 1169.939602] Lustre: server umount lustre-OST0000 complete [ 1186.403620] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 1189.247634] Key type lgssc unregistered [ 1189.556897] LNet: 43818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1189.576328] LNetError: 43818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1189.595315] LNet: Removed LNI 192.168.206.144@tcp [ 1190.577536] Key type .llcrypt unregistered [ 1190.581492] Key type ._llcrypt unregistered [ 1209.164403] Key type ._llcrypt registered [ 1209.166755] Key type .llcrypt registered [ 1209.253387] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 1209.965564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1209.998307] alg: No test for adler32 (adler32-zlib) [ 1210.953690] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 1211.117310] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 1212.771068] Key type lgssc registered [ 1213.703813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1220.774633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1220.788229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1222.182122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1226.320355] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1232.352873] LustreError: 45213:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1232.370268] LustreError: 45213:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1233.508983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1233.792054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1237.472413] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1244.104928] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1248.765374] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1254.365383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1254.703461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1261.139806] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1268.109832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1269.014757] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1277.223026] Lustre: lustre-OST0000: Not available for connect from 192.168.206.44@tcp (stopping) [ 1278.442691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1278.446613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1278.449423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1278.464389] Lustre: Skipped 1 previous similar message [ 1278.492266] Lustre: Skipped 1 previous similar message [ 1282.321217] Lustre: lustre-OST0000: Not available for connect from 192.168.206.44@tcp (stopping) [ 1282.783359] Lustre: server umount lustre-OST0000 complete [ 1300.448069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1300.454533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1300.465753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1304.033595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1305.773850] Lustre: server umount lustre-MDT0000 complete [ 1308.824270] LustreError: 45193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789083959 with bad export cookie 17829315284266668257 [ 1308.825735] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1308.834688] LustreError: 45193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1315.203647] Lustre: server umount lustre-MDT0001 complete [ 1323.702728] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 1326.085722] Key type lgssc unregistered [ 1326.362377] LNet: 48922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.368348] LNetError: 48922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1326.382821] LNet: Removed LNI 192.168.206.144@tcp [ 1327.168589] Key type .llcrypt unregistered [ 1327.171100] Key type ._llcrypt unregistered [ 1345.971722] Key type ._llcrypt registered [ 1345.974358] Key type .llcrypt registered [ 1346.068411] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 1346.979837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1347.051635] alg: No test for adler32 (adler32-zlib) [ 1348.063204] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 1348.240455] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 1349.880744] Key type lgssc registered [ 1350.857305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1358.034139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1358.073248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1359.472510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1363.254911] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1370.082553] LustreError: 50319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1370.097143] LustreError: 50319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1370.501393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1370.781269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1374.641608] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1381.216715] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1386.765950] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1393.169098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1393.565939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1400.394984] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1404.856921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1408.065281] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1414.302583] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1417.974313] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1430.131854] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-1019h47m59s [ 1432.003297] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1435.618288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1435.633517] Lustre: Skipped 2 previous similar messages [ 1435.633536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1435.647198] Lustre: Skipped 1 previous similar message [ 1436.445591] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.44@tcp (stopping) [ 1436.448024] Lustre: Skipped 1 previous similar message [ 1437.513935] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.44@tcp (stopping) [ 1437.526145] Lustre: Skipped 3 previous similar messages [ 1440.277373] Lustre: server umount lustre-MDT0000 complete [ 1442.576341] LustreError: 52808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1443.148410] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1448.731809] LustreError: 52157:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1448.758527] LustreError: 52157:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1450.071812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1450.199424] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1450.548152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1450.623294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1451.566435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1455.834821] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1459.299302] Lustre: DEBUG MARKER: Wait for df (36682) ... [ 1461.036160] Lustre: DEBUG MARKER: done [ 1465.827465] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1465.843104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1465.863489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1466.849065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1471.979169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1471.986275] Lustre: Skipped 2 previous similar messages [ 1479.136037] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1479.366740] Lustre: server umount lustre-OST0000 complete [ 1487.331572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1487.333261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1487.347043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1487.365704] Lustre: Skipped 1 previous similar message [ 1487.373885] Lustre: Skipped 2 previous similar messages [ 1489.526832] Lustre: server umount lustre-MDT0000 complete [ 1492.686881] LustreError: 50299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789084143 with bad export cookie 7023081805786677145 [ 1492.687841] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1492.701740] LustreError: 50299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1492.938249] Lustre: server umount lustre-MDT0001 complete [ 1503.696344] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 1506.422106] Key type lgssc unregistered [ 1506.724704] LNet: 55241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1506.729211] LNetError: 55241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1506.756913] LNet: Removed LNI 192.168.206.144@tcp [ 1507.565456] Key type .llcrypt unregistered [ 1507.575557] Key type ._llcrypt unregistered [ 1540.776417] Key type ._llcrypt registered [ 1540.778178] Key type .llcrypt registered [ 1540.833106] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 1541.471982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1541.487211] alg: No test for adler32 (adler32-zlib) [ 1542.434281] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 1542.560647] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 1544.175157] Key type lgssc registered [ 1544.985330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1551.399632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1551.422717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1552.621897] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1552.644256] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1552.794827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1556.417216] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1564.034267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1564.115939] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1564.144735] Lustre: 57430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1564.424846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1568.177123] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1573.562192] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1577.673249] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1583.177712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1583.287047] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1583.418955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1588.389260] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1590.885105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1594.077666] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1600.673558] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-1019h50m50s [ 1602.140951] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1607.494141] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1609.294944] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1609.350209] LustreError: 59661:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1629.476432] Lustre: lustre-MDT0000: Client 609f7835-90c4-4b11-8ffe-e5efc22e64fb (at 192.168.206.44@tcp) reconnecting [ 1649.407406] LustreError: 59661:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1651.593375] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1653.795457] Lustre: DEBUG MARKER: done [ 1661.256207] Lustre: server umount lustre-OST0000 complete [ 1667.047958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1667.059541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1667.078476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1668.128154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1668.135928] Lustre: Skipped 2 previous similar messages [ 1674.212452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1674.224717] Lustre: Skipped 1 previous similar message [ 1675.296600] Lustre: server umount lustre-MDT0000 complete [ 1678.303219] Lustre: 56751:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789084313/real 1789084313] req@00000000c50c86ec x1875990746811136/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789084329 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1678.324088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1678.336468] Lustre: Skipped 1 previous similar message [ 1678.597061] LustreError: 57422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789084329 with bad export cookie 16063627330697158130 [ 1678.600584] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1678.613930] LustreError: 57422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1678.989557] Lustre: server umount lustre-MDT0001 complete [ 1687.748653] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 1689.837370] Key type lgssc unregistered [ 1690.043892] LNet: 61753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1690.054707] LNetError: 61753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1690.067337] LNet: Removed LNI 192.168.206.144@tcp [ 1690.713316] Key type .llcrypt unregistered [ 1690.716313] Key type ._llcrypt unregistered [ 1729.472285] Key type ._llcrypt registered [ 1729.474540] Key type .llcrypt registered [ 1729.586155] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 1730.845658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1730.913854] alg: No test for adler32 (adler32-zlib) [ 1731.977349] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 1732.195167] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 1733.847178] Key type lgssc registered [ 1734.933361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1737.876554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1737.892378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1739.109888] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1739.137074] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1739.262956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1740.054475] Lustre: Failing over lustre-MDT0000 [ 1740.481331] Lustre: server umount lustre-MDT0000 complete [ 1746.530911] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 19:53:15 (1789084395) [ 1753.780906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1754.403103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1758.741890] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1767.166157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1767.229666] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1767.256403] Lustre: 64530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1767.510249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1771.306469] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1777.730759] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1782.637049] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1788.777646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1788.968181] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1789.174552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1795.296188] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1798.293258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1802.707597] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1808.541843] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1818.598720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1818.611554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1818.620699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1820.463820] Lustre: server umount lustre-MDT0000 complete [ 1823.917937] LustreError: 64523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789084474 with bad export cookie 9075573813619064919 [ 1823.926970] LustreError: 64523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1823.927637] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1824.318195] Lustre: server umount lustre-MDT0001 complete [ 1827.322644] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1831.445821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1840.564834] Lustre: 63453:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789084475/real 1789084475] req@000000007bd7df80 x1875990946027904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789084491 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1840.608104] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1840.624927] Lustre: Skipped 2 previous similar messages [ 1841.165975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1844.639272] Lustre: 63455:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789084479/real 1789084479] req@00000000f7523467 x1875990946028032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789084495 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1848.866417] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7df2ea4c8d9a093a [ 1848.876497] Lustre: MGC192.168.206.144@tcp: Connection restored to 0@lo (at 0@lo) [ 1849.266886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1849.365541] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1850.272657] Lustre: 63453:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789084484/real 1789084484] req@000000007467d77a x1875990946028416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789084500 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1850.316210] Lustre: 63453:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1853.859946] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1854.434895] Lustre: 63454:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789084489/real 1789084489] req@000000000a7f33cf x1875990946028544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789084505 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1855.458286] LustreError: 68301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1855.477891] LustreError: 68301:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 1859.568452] LustreError: 68301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1859.569190] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1859.594252] LustreError: 68301:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1862.593724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1863.093952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1868.158099] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1868.274406] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1876.506898] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1882.159338] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1888.736075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1888.738834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1888.740769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1888.740776] Lustre: Skipped 3 previous similar messages [ 1888.772523] Lustre: Skipped 2 previous similar messages [ 1890.043493] Lustre: server umount lustre-MDT0000 complete [ 1893.199098] LustreError: 66385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789084544 with bad export cookie 9075573813619067194 [ 1893.200638] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1893.214257] LustreError: 66385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1893.553501] Lustre: server umount lustre-MDT0001 complete [ 1899.709458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1908.556625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1909.216244] Lustre: 63455:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789084544/real 1789084544] req@00000000d4623500 x1875990946054400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789084560 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1909.250266] Lustre: 63455:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1909.265239] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1919.540297] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7df2ea4c8d9a0e9d [ 1919.567628] Lustre: MGC192.168.206.144@tcp: Connection restored to 0@lo (at 0@lo) [ 1919.844765] LustreError: 66379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1919.923556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1920.006050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1920.483384] Lustre: 63453:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789084555/real 1789084555] req@0000000034a8180a x1875990946054656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789084571 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1920.527743] Lustre: 63453:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1924.346478] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1925.093087] LustreError: 69037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1928.181884] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1932.267690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1937.223737] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1944.555325] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1950.388738] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1958.368739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1958.392756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1958.418207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1958.422780] Lustre: Skipped 2 previous similar messages [ 1960.423889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1960.444477] Lustre: Skipped 2 previous similar messages [ 1965.537339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1965.549296] Lustre: Skipped 2 previous similar messages [ 1971.167779] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1971.372883] Lustre: server umount lustre-MDT0000 complete [ 1975.229936] LustreError: 66385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789084626 with bad export cookie 9075573813619068573 [ 1975.231967] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1975.238889] LustreError: 66385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1975.776076] LustreError: 70984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1975.800972] LustreError: 70984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1975.801179] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1975.836118] Lustre: Skipped 2 previous similar messages [ 1976.067668] Lustre: server umount lustre-MDT0001 complete [ 1983.181255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1987.235771] Lustre: DEBUG MARKER: files compared the same [ 1989.759290] Lustre: server umount lustre-OST0000 complete [ 2007.112831] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 2010.485947] Key type lgssc unregistered [ 2010.886612] LNet: 74261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2010.906822] LNetError: 74261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2010.933830] LNet: Removed LNI 192.168.206.144@tcp [ 2012.103290] Key type .llcrypt unregistered [ 2012.107061] Key type ._llcrypt unregistered [ 2039.044838] Key type ._llcrypt registered [ 2039.048129] Key type .llcrypt registered [ 2039.133943] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 2040.035566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2040.092282] alg: No test for adler32 (adler32-zlib) [ 2041.104648] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 2041.342857] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 2043.001542] Key type lgssc registered [ 2044.196693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2052.004853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2052.019560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2053.443771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2057.955219] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug +malloc all [ 2063.842156] LustreError: 75679:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2063.869584] LustreError: 75679:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2066.289356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2066.583848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2070.959813] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug +malloc all [ 2078.492744] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2083.763289] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2090.786062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2091.250218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.395432] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug +malloc all [ 2098.429698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2105.478329] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2114.016213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2114.027331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2114.033123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2114.053779] Lustre: Skipped 1 previous similar message [ 2114.069139] Lustre: Skipped 1 previous similar message [ 2119.136880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2119.833169] Lustre: server umount lustre-OST0000 complete [ 2124.257161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2124.288952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2124.303247] Lustre: Skipped 2 previous similar messages [ 2129.376943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2129.383551] Lustre: Skipped 1 previous similar message [ 2134.496435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2134.500910] Lustre: Skipped 1 previous similar message [ 2138.079132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2138.312622] Lustre: server umount lustre-MDT0000 complete [ 2139.619035] LustreError: 76406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.637543] LustreError: 76406:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2141.991953] LustreError: 75659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789084792 with bad export cookie 16130156957263494433 [ 2141.996900] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2142.013776] LustreError: 75659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2142.244859] Lustre: server umount lustre-MDT0001 complete [ 2151.953344] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 2154.637128] Key type lgssc unregistered [ 2155.018684] LNet: 79210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2155.030411] LNetError: 79210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2155.053518] LNet: Removed LNI 192.168.206.144@tcp [ 2155.893208] Key type .llcrypt unregistered [ 2155.895462] Key type ._llcrypt unregistered [ 2177.842461] Key type ._llcrypt registered [ 2177.845288] Key type .llcrypt registered [ 2178.001745] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 2178.670541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2178.683686] alg: No test for adler32 (adler32-zlib) [ 2179.792451] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 2180.010485] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 2181.655175] Key type lgssc registered [ 2182.704688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2189.839823] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2189.865540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2205.407389] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2210.527475] LustreError: 80602:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.144@tcp: failed processing log, type 4: rc = -110 [ 2237.279420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2244.663963] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2252.203163] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2259.609620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.796165] Lustre: *** cfs_fail_loc=706, val=0*** [ 2260.235962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2260.310856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2264.682336] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2265.570698] LustreError: 80627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2265.586525] LustreError: 80627:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2270.693231] LustreError: 80628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2271.724402] LustreError: 81703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2271.759508] LustreError: 81703:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2273.630143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2273.948982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2277.590496] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2284.322858] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2289.911551] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2294.241185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2294.252197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.267211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2294.755281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.778052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2298.079842] Lustre: server umount lustre-OST0000 complete [ 2304.481300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2304.496743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2304.510428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2308.207606] Lustre: server umount lustre-MDT0000 complete [ 2310.118234] LustreError: 81706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.144707] LustreError: 81706:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2312.455033] LustreError: 80610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789084963 with bad export cookie 13453126334895620290 [ 2312.458391] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2312.474090] LustreError: 80610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2312.768632] Lustre: server umount lustre-MDT0001 complete [ 2324.119046] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 2327.258740] Key type lgssc unregistered [ 2327.541383] LNet: 84230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2327.549605] LNetError: 84230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2327.565909] LNet: Removed LNI 192.168.206.144@tcp [ 2328.259829] Key type .llcrypt unregistered [ 2328.264359] Key type ._llcrypt unregistered [ 2350.715492] Key type ._llcrypt registered [ 2350.717144] Key type .llcrypt registered [ 2350.846691] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 2351.583813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2351.692184] alg: No test for adler32 (adler32-zlib) [ 2352.856371] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 2353.153098] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 2354.903694] Key type lgssc registered [ 2355.861647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2362.950985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2362.970954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2368.204398] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2375.609130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2376.193959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2380.641353] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2381.283548] LustreError: 86338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.311648] LustreError: 86338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2386.402098] LustreError: 86342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.422863] LustreError: 86342:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2389.370702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2389.652352] LustreError: 86338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2389.667331] LustreError: 86338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2389.688607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2395.104884] LustreError: 86341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.133978] LustreError: 86341:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2395.753919] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2400.225949] LustreError: 86342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2400.242904] LustreError: 86342:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2403.768193] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2403.789519] Lustre: MGS: Client f6366fb9-f0d9-49e5-ab63-96af05bf036e (at 0@lo) reconnecting [ 2403.831847] LustreError: 86352:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2404.119555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2404.197663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2408.773232] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2424.806715] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.811812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2424.825063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2424.841848] LustreError: Skipped 1 previous similar message [ 2427.861984] Lustre: server umount lustre-OST0000 complete [ 2435.042955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2435.046478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2435.064645] Lustre: Skipped 1 previous similar message [ 2435.065375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2435.082210] Lustre: Skipped 2 previous similar messages [ 2437.627035] Lustre: server umount lustre-MDT0000 complete [ 2440.163025] LustreError: 86338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.177544] LustreError: 86338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2441.150807] Lustre: server umount lustre-MDT0001 complete [ 2444.332907] Lustre: server umount MGS complete [ 2455.194499] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 20:05:05 (1789085105) [ 2479.833463] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 2489.304263] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 2519.193374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2523.565989] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2532.583108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2532.979965] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2533.038421] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2533.127763] Lustre: lustre-MDT0001: new disk, initializing [ 2533.250058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2533.278782] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2533.286930] Lustre: 93504:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2533.298065] LustreError: 93504:0:(lod_dev.c:512:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2537.858438] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2547.791428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2548.008743] Lustre: lustre-OST0000: new disk, initializing [ 2548.012349] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2548.018248] Lustre: 94269:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2548.094747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2554.761428] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2555.493130] Lustre: 94588:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000919086f9 x1875991597194880/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789085222 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2564.500689] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2570.757886] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2570.770035] Lustre: MGS: Client f1c3a06e-590a-45f0-81b6-430676067bd1 (at 0@lo) reconnecting [ 2570.803767] Lustre: 92772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2570.816484] LustreError: 93490:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2570.827021] LustreError: 93490:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2570.869317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2570.942538] Lustre: lustre-MDT0000: new disk, initializing [ 2571.117132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2571.149172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2571.194689] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2572.257290] Lustre: 95341:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000410037a6 x1875991597201792/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789085239 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2574.992651] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2577.440477] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2581.357501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2581.363643] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2581.534547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2607.584338] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2607.599486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2607.641994] Lustre: Skipped 1 previous similar message [ 2607.679330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2611.845251] Lustre: server umount lustre-OST0000 complete [ 2619.874058] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2619.875966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2619.876697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2619.876702] Lustre: Skipped 3 previous similar messages [ 2619.880435] LustreError: Skipped 1 previous similar message [ 2619.916165] Lustre: Skipped 2 previous similar messages [ 2622.702487] Lustre: server umount lustre-MDT0000 complete [ 2624.992529] LustreError: 95721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2625.012574] LustreError: 95721:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2632.787831] Lustre: server umount lustre-MDT0001 complete [ 2636.062165] Lustre: server umount MGS complete [ 2645.622778] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 20:08:15 (1789085295) [ 2659.471073] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 2661.698819] Key type lgssc unregistered [ 2661.986503] LNet: 97838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2661.990932] LNetError: 97838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2662.019606] LNet: Removed LNI 192.168.206.144@tcp [ 2662.733364] Key type .llcrypt unregistered [ 2662.738385] Key type ._llcrypt unregistered [ 2676.426590] Key type ._llcrypt registered [ 2676.428987] Key type .llcrypt registered [ 2676.530983] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 2677.541036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2677.566974] alg: No test for adler32 (adler32-zlib) [ 2678.631742] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 2678.842464] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 2680.487174] Key type lgssc registered [ 2681.348507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2688.004702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2688.018240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2689.322658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2693.384199] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2699.749282] LustreError: 99042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2699.773081] LustreError: 99042:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2701.478953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2701.977780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2706.300836] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2712.924518] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2717.767568] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2723.473405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2723.908487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2730.794219] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2734.143506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2738.859964] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2749.414242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2749.421515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2749.436077] Lustre: Skipped 1 previous similar message [ 2749.451751] Lustre: Skipped 1 previous similar message [ 2753.718411] Lustre: server umount lustre-OST0000 complete [ 2758.117522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2758.132544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.161204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2759.651029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2759.652726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2763.677248] Lustre: server umount lustre-MDT0000 complete [ 2764.769934] LustreError: 100095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2764.798167] LustreError: 100095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2767.324222] LustreError: 99022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789085418 with bad export cookie 11388006816641001481 [ 2767.328562] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2767.338649] LustreError: 99022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2767.653551] Lustre: server umount lustre-MDT0001 complete [ 2777.559193] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 2779.945596] Key type lgssc unregistered [ 2780.227465] LNet: 102573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2780.236652] LNetError: 102573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2780.267355] LNet: Removed LNI 192.168.206.144@tcp [ 2781.046127] Key type .llcrypt unregistered [ 2781.050935] Key type ._llcrypt unregistered [ 2795.646686] Key type ._llcrypt registered [ 2795.648909] Key type .llcrypt registered [ 2795.749883] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 2796.834081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2796.854395] alg: No test for adler32 (adler32-zlib) [ 2797.996123] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 2798.265231] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 2799.959332] Key type lgssc registered [ 2800.912934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2806.261463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2806.271097] LustreError: 103652:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2806.286503] LustreError: 103656:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2806.292573] LustreError: 103652:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4984 [ 2806.300607] LustreError: 103652:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2806.309645] LustreError: 103652:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2806.316755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2807.831977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2815.246689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2815.514819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2819.486368] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2824.631564] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing lsmod [ 2827.527093] LustreError: 104863:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2827.533183] LustreError: 104862:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2827.552108] LustreError: 104863:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4987 [ 2827.557582] LustreError: 104863:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2827.572562] LustreError: 104863:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2827.582189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2828.077912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2830.826448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2830.827855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2830.851796] Lustre: Skipped 1 previous similar message [ 2830.852190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2835.941539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2835.948695] Lustre: Skipped 3 previous similar messages [ 2841.066629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2841.081018] Lustre: Skipped 3 previous similar messages [ 2845.151143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2845.334893] Lustre: server umount lustre-MDT0000 complete [ 2846.183234] LustreError: 103688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2846.215283] LustreError: 103688:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2849.208105] LustreError: 103670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789085500 with bad export cookie 2648153926047630001 [ 2849.219406] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2849.232828] LustreError: 103670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2849.311238] LustreError: 104126:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d7cdebfb x1875992063791360/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2849.565965] Lustre: server umount lustre-MDT0001 complete [ 2862.620867] Lustre: server umount lustre-OST0000 complete [ 2871.232943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2871.696201] LustreError: 105931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2871.776244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2875.972213] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2876.902862] LustreError: 105932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2882.017355] LustreError: 105931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2885.114468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2885.617871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2890.098662] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2898.278795] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2904.927786] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2913.454096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2913.980867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2917.075876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2921.510453] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2929.238515] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2937.837276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2937.853352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2937.869146] Lustre: Skipped 1 previous similar message [ 2937.882143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2940.900712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2940.903492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2944.049542] Lustre: server umount lustre-OST0000 complete [ 2951.151941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2951.153874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2951.162446] Lustre: Skipped 1 previous similar message [ 2951.179489] Lustre: Skipped 3 previous similar messages [ 2955.269118] Lustre: server umount lustre-MDT0000 complete [ 2956.257241] LustreError: 105932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2956.318800] LustreError: 105932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2959.432615] LustreError: 105912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789085610 with bad export cookie 2648153926047630680 [ 2959.447354] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2959.452279] LustreError: 105912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2959.988852] Lustre: server umount lustre-MDT0001 complete [ 2970.967335] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 2973.901586] Key type lgssc unregistered [ 2974.176239] LNet: 109464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2974.183918] LNetError: 109464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2974.200506] LNet: Removed LNI 192.168.206.144@tcp [ 2975.216569] Key type .llcrypt unregistered [ 2975.219208] Key type ._llcrypt unregistered [ 2998.882455] Key type ._llcrypt registered [ 2998.884674] Key type .llcrypt registered [ 2998.990725] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 3000.125916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3000.211167] alg: No test for adler32 (adler32-zlib) [ 3001.301764] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 3001.581590] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 3003.303165] Key type lgssc registered [ 3004.722275] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3012.843932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3012.865238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3014.295581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3019.026403] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3024.870668] LustreError: 110882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3024.885024] LustreError: 110882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3027.487293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3027.886223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3032.032749] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3038.929270] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3045.570509] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3052.973783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3053.247716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3059.379337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3059.758288] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3068.013294] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3077.439350] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3081.759921] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3088.360249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3088.371677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3088.378058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3093.051227] Lustre: server umount lustre-OST0000 complete [ 3098.593857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3098.607342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3098.616397] Lustre: Skipped 2 previous similar messages [ 3098.623134] Lustre: Skipped 1 previous similar message [ 3103.706690] Lustre: server umount lustre-MDT0000 complete [ 3103.716787] LustreError: 110882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3103.739975] LustreError: 110882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3108.119440] LustreError: 110862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789085759 with bad export cookie 9593794173359330590 [ 3108.126872] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3108.140395] LustreError: 110862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3108.532717] Lustre: server umount lustre-MDT0001 complete [ 3121.262073] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 3124.014171] Key type lgssc unregistered [ 3124.556797] LNet: 114520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3124.571919] LNetError: 114520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3124.604177] LNet: Removed LNI 192.168.206.144@tcp [ 3125.765487] Key type .llcrypt unregistered [ 3125.771960] Key type ._llcrypt unregistered [ 3146.999078] Key type ._llcrypt registered [ 3147.002708] Key type .llcrypt registered [ 3147.198631] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 3149.346476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3149.379064] alg: No test for adler32 (adler32-zlib) [ 3150.427158] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 3150.688508] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 3152.399167] Key type lgssc registered [ 3153.795686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3163.183847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3163.230397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3164.765317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3171.229266] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3174.885443] LustreError: 115743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3174.904482] LustreError: 115743:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3180.016252] LustreError: 115742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3181.431766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3182.023179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3187.356374] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3195.868243] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3202.688989] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3210.317643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3210.793149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3218.044392] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3228.374731] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3234.373250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3244.517192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3244.517587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3244.533295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3244.567549] Lustre: Skipped 1 previous similar message [ 3249.216545] Lustre: server umount lustre-OST0000 complete [ 3253.728391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3253.733446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3253.743274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3253.745859] Lustre: Skipped 1 previous similar message [ 3254.754610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3254.755601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3254.785877] Lustre: Skipped 1 previous similar message [ 3259.456907] Lustre: server umount lustre-MDT0000 complete [ 3259.873399] LustreError: 115743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3259.889822] LustreError: 115743:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3263.612119] LustreError: 115723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789085914 with bad export cookie 5116227629354284407 [ 3263.622251] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3263.627924] LustreError: 115723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3264.025268] Lustre: server umount lustre-MDT0001 complete [ 3276.493842] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 3279.110550] Key type lgssc unregistered [ 3279.403666] LNet: 119300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3279.421635] LNetError: 119300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3280.485934] LNet: Removed LNI 192.168.206.144@tcp [ 3281.312182] Key type .llcrypt unregistered [ 3281.313876] Key type ._llcrypt unregistered [ 3307.239594] Key type ._llcrypt registered [ 3307.242421] Key type .llcrypt registered [ 3307.397383] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 3308.585766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3308.845246] alg: No test for adler32 (adler32-zlib) [ 3310.006791] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 3310.313828] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 3312.039207] Key type lgssc registered [ 3313.448523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3322.887413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3322.907576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3324.647984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3330.703701] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3334.118286] LustreError: 120720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3334.143887] LustreError: 120720:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3339.239307] LustreError: 120721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3340.307933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3340.769038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3346.046357] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3353.988741] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3360.478328] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3368.379891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3368.777269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3369.799944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3375.943762] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3384.586189] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3392.651386] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3396.773306] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3396.779245] Lustre: Skipped 2 previous similar messages [ 3401.963405] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3401.970204] Lustre: Skipped 2 previous similar messages [ 3402.897808] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3402.974856] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3413.811556] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.44@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3413.823478] Lustre: Skipped 2 previous similar messages [ 3422.827106] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.44@tcp 192.168.0.[2,10]@tcp [ 3427.674955] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.44/24@tcp in log lustre-MDT0000 [ 3427.688359] Lustre: Skipped 2 previous similar messages [ 3429.662315] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.44/24@tcp [ 3429.670266] Lustre: Skipped 1 previous similar message [ 3446.752232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3446.763200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3446.773029] Lustre: Skipped 1 previous similar message [ 3446.778382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3446.803702] Lustre: Skipped 1 previous similar message [ 3449.972139] Lustre: server umount lustre-OST0000 complete [ 3456.995442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3457.001676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3457.010733] Lustre: Skipped 1 previous similar message [ 3457.018179] Lustre: Skipped 1 previous similar message [ 3460.514723] Lustre: server umount lustre-MDT0000 complete [ 3462.119205] LustreError: 120720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3462.138319] LustreError: 120720:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3464.794706] LustreError: 120700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789086115 with bad export cookie 7535919651548552016 [ 3464.800770] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3464.810996] LustreError: 120700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3465.208588] Lustre: server umount lustre-MDT0001 complete [ 3477.054094] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 3480.088551] Key type lgssc unregistered [ 3480.473710] LNet: 125530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3480.481508] LNetError: 125530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3480.502932] LNet: Removed LNI 192.168.206.144@tcp [ 3481.469895] Key type .llcrypt unregistered [ 3481.476486] Key type ._llcrypt unregistered [ 3512.799782] Key type ._llcrypt registered [ 3512.802797] Key type .llcrypt registered [ 3512.921388] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 3514.165600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3514.199623] alg: No test for adler32 (adler32-zlib) [ 3515.439732] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 3515.626515] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 3517.295162] Key type lgssc registered [ 3518.374612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3526.474903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3526.517499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3528.053487] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3528.081793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3528.090875] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.44/24@tcp [ 3532.446437] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3537.384344] LustreError: 127094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3537.405288] LustreError: 127094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3541.782483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3542.146372] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3542.177438] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.44/24@tcp [ 3542.221933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3547.945339] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3557.907313] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3567.102824] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3575.993358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3576.611179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3584.498612] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3593.395333] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3598.124663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3608.548472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3608.561298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3608.562242] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3608.564373] Lustre: Skipped 1 previous similar message [ 3608.587641] Lustre: Skipped 1 previous similar message [ 3613.130158] Lustre: server umount lustre-OST0000 complete [ 3618.789014] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3618.793183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3618.809972] Lustre: Skipped 1 previous similar message [ 3618.828146] Lustre: Skipped 1 previous similar message [ 3623.634973] Lustre: server umount lustre-MDT0000 complete [ 3623.907201] LustreError: 127088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3623.925419] LustreError: 127088:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3627.908077] LustreError: 127073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789086278 with bad export cookie 16706061842035614774 [ 3627.914347] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3627.919357] LustreError: 127073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3628.436768] Lustre: server umount lustre-MDT0001 complete [ 3639.028517] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 3642.218667] Key type lgssc unregistered [ 3642.505542] LNet: 130683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3642.512153] LNetError: 130683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3642.533200] LNet: Removed LNI 192.168.206.144@tcp [ 3643.468695] Key type .llcrypt unregistered [ 3643.470830] Key type ._llcrypt unregistered [ 3672.018780] Key type ._llcrypt registered [ 3672.021466] Key type .llcrypt registered [ 3672.118573] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 3691.696043] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 3692.733487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3692.775650] alg: No test for adler32 (adler32-zlib) [ 3694.138732] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 3694.411062] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 3696.183182] Key type lgssc registered [ 3697.619067] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3733.967805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3734.001327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3735.347269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3735.367141] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3735.420325] Lustre: lustre-MDT0000: new disk, initializing [ 3735.487374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3735.508601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3740.109588] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3751.239884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3751.415586] Lustre: 134086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3751.500318] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3751.503965] Lustre: Skipped 1 previous similar message [ 3751.605773] Lustre: lustre-MDT0001: new disk, initializing [ 3751.722236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3751.766124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3751.776634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3757.343384] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3769.159237] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3774.648562] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3782.101311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3782.604784] Lustre: lustre-OST0000: new disk, initializing [ 3782.613968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3782.625285] Lustre: 136055:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3782.744068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3788.329948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3788.335672] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3788.451785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3789.296851] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3798.695639] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3803.196205] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3803.385894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3807.538890] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3807.756461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3813.856058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3813.858116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3813.865923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3813.879849] Lustre: Skipped 1 previous similar message [ 3813.896487] Lustre: Skipped 1 previous similar message [ 3815.735700] Lustre: server umount lustre-OST0000 complete [ 3823.586864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3823.593359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3823.609546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3825.582798] Lustre: server umount lustre-MDT0000 complete [ 3829.116017] LustreError: 134076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789086480 with bad export cookie 1687040709794934440 [ 3829.120991] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3829.130291] LustreError: 134076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3829.401876] Lustre: server umount lustre-MDT0001 complete [ 3836.635098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3837.054711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3841.010969] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3842.531509] LustreError: 138272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3842.548307] LustreError: 138272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3846.628409] LustreError: 138273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3848.959911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3849.206934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3853.338290] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3860.163305] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3866.842515] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3874.437061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3874.821844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3881.047202] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3888.551209] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3893.109926] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3893.281229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3897.772907] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3898.033909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3906.701563] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3906.855461] Lustre: lustre-OST0001: new disk, initializing [ 3906.858647] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3906.864646] Lustre: 141644:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3906.946407] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3912.639339] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3913.281500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3913.292547] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3913.402278] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3920.263296] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3920.453364] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3924.848550] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3925.067856] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3930.194701] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8bcfd8343000.ost_server_uuid 50 [ 3931.904944] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8bcfd8343000.ost_server_uuid in FULL state after 0 sec [ 3943.866146] Lustre: server umount lustre-OST0001 complete [ 3944.416088] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3944.418949] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3944.421265] LustreError: Skipped 1 previous similar message [ 3944.423417] LustreError: 141367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3944.451609] Lustre: Skipped 2 previous similar messages [ 3948.512674] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3948.521822] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3948.534572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3948.544232] Lustre: Skipped 2 previous similar messages [ 3949.539886] LustreError: 141367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3949.557675] LustreError: 141367:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3954.659150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3954.660389] LustreError: 140093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3954.673688] Lustre: Skipped 3 previous similar messages [ 3954.683852] LustreError: 140093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3959.779616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3962.335775] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3962.477914] Lustre: server umount lustre-OST0000 complete [ 3966.945335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3966.952943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3966.964113] Lustre: Skipped 1 previous similar message [ 3970.018171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3970.026437] Lustre: Skipped 3 previous similar messages [ 3981.279128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3981.438837] Lustre: server umount lustre-MDT0000 complete [ 3984.685612] LustreError: 138252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789086635 with bad export cookie 1687040709794935322 [ 3984.688512] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3984.695894] LustreError: 138252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3985.038134] Lustre: server umount lustre-MDT0001 complete [ 4006.897600] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 4010.485794] Key type lgssc unregistered [ 4010.941661] LNet: 144647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4010.980853] LNetError: 144647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4011.000295] LNet: Removed LNI 192.168.206.144@tcp [ 4011.955462] Key type .llcrypt unregistered [ 4011.958846] Key type ._llcrypt unregistered [ 4059.546435] Key type ._llcrypt registered [ 4059.555988] Key type .llcrypt registered [ 4059.711647] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 4078.390368] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 4080.000691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4080.028746] alg: No test for adler32 (adler32-zlib) [ 4081.120370] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 4081.394336] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 4083.144195] Key type lgssc registered [ 4084.730961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4122.601412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4122.631509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4123.951930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4123.975869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4124.049384] Lustre: lustre-MDT0000: new disk, initializing [ 4124.116355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4124.147215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4128.885397] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4139.844105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4140.000330] Lustre: 148806:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4140.058820] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4140.062373] Lustre: Skipped 1 previous similar message [ 4140.147485] Lustre: lustre-MDT0001: new disk, initializing [ 4140.230572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4140.299825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4140.314194] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4144.939524] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4155.570753] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4163.177749] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4172.076178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4172.497156] Lustre: lustre-OST0000: new disk, initializing [ 4172.514737] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4172.521995] Lustre: 150777:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4172.657056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4180.680124] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4181.647303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4181.655424] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4181.820526] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4193.001112] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4198.077741] Lustre: Failing over lustre-OST0000 [ 4198.260208] Lustre: server umount lustre-OST0000 complete [ 4215.270511] Lustre: 147030:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789086850/real 1789086850] req@000000000a5b78e9 x1875993409129216/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789086866 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4215.310872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4217.846542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4218.165801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4218.176798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4218.199778] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4219.169881] Lustre: 147032:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789086853/real 1789086853] req@000000007c09db71 x1875993409129600/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789086869 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4220.201308] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4223.337316] Lustre: 147029:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789086858/real 1789086858] req@00000000855fe3c9 x1875993409131264/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789086874 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4223.362844] Lustre: 147029:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4224.324298] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4226.876676] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [ 4226.879651] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4228.357036] Lustre: 147031:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789086863/real 1789086863] req@0000000061de0a55 x1875993409132032/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789086879 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4228.372928] Lustre: Failing over lustre-MDT0000 [ 4228.377836] Lustre: 147031:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4228.781257] Lustre: server umount lustre-MDT0000 complete [ 4230.626968] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4230.647311] Lustre: Skipped 2 previous similar messages [ 4235.748960] LustreError: 151497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4235.769523] LustreError: 151497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4237.077705] LustreError: 148813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4240.867834] LustreError: 148813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4240.889809] LustreError: 148813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4245.992089] LustreError: 148817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4246.018015] LustreError: 148817:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4247.008490] Lustre: 147032:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789086881/real 1789086881] req@00000000483b825b x1875993409142016/t0(0) o400->MGC192.168.206.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789086897 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4247.049379] Lustre: 147032:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4247.057018] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4247.854274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4252.134770] LustreError: 148817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4252.149559] LustreError: 148817:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4256.530551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4256.580944] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4257.552802] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4260.818866] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4261.879241] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4261.894028] Lustre: Skipped 1 previous similar message [ 4261.925990] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4261.967101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4272.135460] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4272.151577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4272.165902] Lustre: Skipped 1 previous similar message [ 4275.564745] Lustre: server umount lustre-OST0000 complete [ 4282.336829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4282.337926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4282.351878] Lustre: Skipped 2 previous similar messages [ 4282.360395] Lustre: Skipped 1 previous similar message [ 4285.673067] Lustre: server umount lustre-MDT0000 complete [ 4287.459562] LustreError: 148812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4287.497926] LustreError: 148812:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4289.457158] LustreError: 148799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789086940 with bad export cookie 615013317576167318 [ 4289.464833] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4289.470618] LustreError: 148799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4289.866929] Lustre: server umount lustre-MDT0001 complete [ 4301.489602] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 4304.148681] Key type lgssc unregistered [ 4304.520761] LNet: 154435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4304.532374] LNetError: 154435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4304.547916] LNet: Removed LNI 192.168.206.144@tcp [ 4305.403264] Key type .llcrypt unregistered [ 4305.408858] Key type ._llcrypt unregistered [ 4342.575504] Key type ._llcrypt registered [ 4342.581155] Key type .llcrypt registered [ 4342.725996] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 4343.638973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4343.651969] alg: No test for adler32 (adler32-zlib) [ 4344.752044] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 4345.034253] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 4346.727453] Key type lgssc registered [ 4347.736832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4355.978673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4356.003235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4357.455349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4362.069793] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4367.841509] LustreError: 155971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4367.862036] LustreError: 155971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4372.474777] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4372.961112] LustreError: 155970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4377.048710] LustreError: 155999:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4377.063418] LustreError: 155999:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4377.075261] LustreError: 155999:0:(lod_dev.c:512:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4377.562801] Lustre: server umount lustre-MDT0000 complete [ 4395.495490] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 4405.513837] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 4449.411807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4449.635645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4449.660925] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4449.768679] Lustre: lustre-MDT0000: new disk, initializing [ 4449.868128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4449.896806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4454.490676] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4465.390440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4465.551774] Lustre: 159301:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4465.606291] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4465.610219] Lustre: Skipped 1 previous similar message [ 4465.686702] Lustre: lustre-MDT0001: new disk, initializing [ 4465.798841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4465.835396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4465.844636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4470.423261] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4481.234964] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4488.257353] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4495.371549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4495.650853] Lustre: lustre-OST0000: new disk, initializing [ 4495.656091] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4495.668544] Lustre: 161274:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4495.794702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4502.264401] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4503.076435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4503.092227] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4503.229838] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4511.761897] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4516.925741] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4519.247144] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 20:39:28 (1789087168) [ 4793.824681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4793.833268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4793.841616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4795.368991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4795.370855] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4795.377068] Lustre: Skipped 1 previous similar message [ 4795.386533] Lustre: Skipped 1 previous similar message [ 4797.383348] Lustre: server umount lustre-MDT0000 complete [ 4800.480835] LustreError: 159310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4800.496870] LustreError: 159310:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4801.507735] LustreError: 161272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789087452 with bad export cookie 1311585727690541602 [ 4801.512538] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4801.526232] LustreError: 161272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4805.599976] LustreError: 159309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4805.601268] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4805.639523] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4805.649413] Lustre: Skipped 1 previous similar message [ 4808.070860] Lustre: server umount lustre-MDT0001 complete [ 4819.316123] Lustre: server umount lustre-OST0000 complete [ 4830.824761] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 4843.510775] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 4884.752528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4885.372138] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4885.416268] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4885.536157] Lustre: lustre-MDT0000: new disk, initializing [ 4885.641036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4885.664497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4892.724186] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4909.213082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4909.572767] Lustre: 165949:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4909.770389] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4909.796805] Lustre: Skipped 1 previous similar message [ 4910.002878] Lustre: lustre-MDT0001: new disk, initializing [ 4910.296399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4910.388980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4910.406508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4916.116248] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4928.448814] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4935.503695] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4944.341765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4944.687878] Lustre: lustre-OST0000: new disk, initializing [ 4944.691642] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4944.704434] Lustre: 167918:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4944.801863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4950.570914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4950.588726] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4950.725648] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4951.600944] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4961.375324] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4965.953656] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4966.295621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4972.347915] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4972.633452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4976.612685] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4976.622767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4976.637106] LustreError: Skipped 1 previous similar message [ 4976.641696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4976.652019] Lustre: Skipped 1 previous similar message [ 4976.655264] Lustre: Skipped 1 previous similar message [ 4981.538170] Lustre: server umount lustre-OST0000 complete [ 4986.849467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4986.851687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4986.857383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4986.857395] Lustre: Skipped 1 previous similar message [ 4986.864848] Lustre: Skipped 1 previous similar message [ 4992.373342] Lustre: server umount lustre-MDT0000 complete [ 4996.561313] LustreError: 165940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789087647 with bad export cookie 1311585727690700432 [ 4996.580411] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4996.587119] LustreError: 165940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4997.101090] Lustre: server umount lustre-MDT0001 complete [ 5005.989186] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:47:35 (1789087655) [ 5025.697732] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 5037.723668] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 5072.915921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5073.150770] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5073.190357] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5073.271486] Lustre: lustre-MDT0000: new disk, initializing [ 5073.335368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5073.352588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5077.097414] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5087.630234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5087.847042] Lustre: 172672:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5088.071211] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5093.342286] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5104.972761] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5112.009178] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5119.156340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5119.521437] Lustre: lustre-OST0000: new disk, initializing [ 5119.523539] Lustre: Skipped 1 previous similar message [ 5119.526785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5119.529662] Lustre: Skipped 2 previous similar messages [ 5119.533531] Lustre: 174653:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5126.329554] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5127.747303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5127.769130] Lustre: Skipped 1 previous similar message [ 5127.781493] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5127.912219] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5138.008861] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5148.654236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5148.659664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5148.668825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5148.668833] Lustre: Skipped 3 previous similar messages [ 5148.700392] Lustre: Skipped 1 previous similar message [ 5152.649560] Lustre: server umount lustre-OST0000 complete [ 5163.566130] Lustre: server umount lustre-MDT0000 complete [ 5164.004331] LustreError: 172680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5164.037369] LustreError: 172680:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5168.042729] LustreError: 172666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789087818 with bad export cookie 1311585727690701314 [ 5168.043986] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5168.052044] LustreError: 172666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5168.371886] Lustre: server umount lustre-MDT0001 complete [ 5177.792374] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:50:27 (1789087827) [ 5193.502717] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 5203.970413] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 5238.287439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5238.474422] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5238.495935] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5238.574519] Lustre: lustre-MDT0000: new disk, initializing [ 5238.686287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5238.693830] Lustre: Skipped 2 previous similar messages [ 5238.710537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5243.177654] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5253.165744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5253.345562] Lustre: 178978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5253.436352] LustreError: 178988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5253.650294] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5257.689247] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5267.132054] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5273.136620] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5280.137990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5280.543423] Lustre: 180956:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5287.524929] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5288.574624] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5288.824234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5297.325893] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5309.416573] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5309.418726] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5309.428045] LustreError: Skipped 1 previous similar message [ 5309.434535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5309.451705] Lustre: Skipped 3 previous similar messages [ 5309.468139] Lustre: Skipped 4 previous similar messages [ 5311.054267] Lustre: server umount lustre-OST0000 complete [ 5329.379721] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5329.680952] Lustre: server umount lustre-MDT0000 complete [ 5329.900477] LustreError: 180097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5333.661598] LustreError: 178969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789087984 with bad export cookie 1311585727690702399 [ 5333.664665] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5333.674254] LustreError: 178969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5334.055611] Lustre: server umount lustre-MDT0001 complete [ 5344.930896] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 5347.914871] Key type lgssc unregistered [ 5348.156947] LNet: 182815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5348.163047] LNetError: 182815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5348.176593] LNet: Removed LNI 192.168.206.144@tcp [ 5349.169191] Key type .llcrypt unregistered [ 5349.172787] Key type ._llcrypt unregistered [ 5373.517645] Key type ._llcrypt registered [ 5373.519890] Key type .llcrypt registered [ 5373.614362] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 5374.717665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5374.879848] alg: No test for adler32 (adler32-zlib) [ 5375.924460] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 5376.232308] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 5377.924041] Key type lgssc registered [ 5379.327467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5387.242959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5387.262967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5388.727734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5393.624718] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5399.012679] LustreError: 184230:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5399.036139] LustreError: 184230:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5402.309581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5402.625014] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5406.925124] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5414.375601] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5419.962944] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5426.441018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5426.769880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5432.326076] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5433.964707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5439.142041] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5458.912633] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5458.924174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5458.932320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5459.936933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5459.937989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5459.985024] Lustre: Skipped 1 previous similar message [ 5462.322779] Lustre: server umount lustre-OST0000 complete [ 5469.168234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5469.175185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5469.200928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5472.279713] Lustre: server umount lustre-MDT0000 complete [ 5475.305347] LustreError: 184231:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5475.321343] LustreError: 184231:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5476.100426] LustreError: 184210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789088127 with bad export cookie 17011723338553883551 [ 5476.107846] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5476.122632] LustreError: 184210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5476.449288] Lustre: server umount lustre-MDT0001 complete [ 5487.080584] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 5489.672631] Key type lgssc unregistered [ 5489.956869] LNet: 187786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5489.966056] LNetError: 187786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5489.980382] LNet: Removed LNI 192.168.206.144@tcp [ 5490.923431] Key type .llcrypt unregistered [ 5490.926460] Key type ._llcrypt unregistered [ 5516.172438] Key type ._llcrypt registered [ 5516.177606] Key type .llcrypt registered [ 5516.280694] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 5517.198238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5517.242161] alg: No test for adler32 (adler32-zlib) [ 5518.370737] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 5518.626917] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 5520.327254] Key type lgssc registered [ 5521.718948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5529.383136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5529.398386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5530.777861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5534.519581] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5540.324648] LustreError: 189205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5540.335618] LustreError: 189205:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5543.481536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5543.948990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5548.159206] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5555.964220] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5562.194583] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5569.651430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5570.164587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5575.300253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5576.059109] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5585.718540] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5596.132316] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5596.152888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5596.183618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5597.990297] Lustre: lustre-OST0000: Not available for connect from 192.168.206.44@tcp (stopping) [ 5599.212428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5599.219598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5599.239764] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5601.612286] Lustre: server umount lustre-OST0000 complete [ 5609.179502] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8bcfc5544000.ost_server_uuid 50 [ 5611.078799] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5625.834464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5625.848771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5625.885966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5625.894973] Lustre: Skipped 1 previous similar message [ 5629.920239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5629.924164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5629.945229] Lustre: Skipped 1 previous similar message [ 5631.537855] Lustre: server umount lustre-MDT0000 complete [ 5635.041291] LustreError: 191334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5635.057211] LustreError: 191334:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5635.675458] LustreError: 189184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789088286 with bad export cookie 256699282677245096 [ 5635.691867] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5635.696210] LustreError: 189184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5636.457294] Lustre: server umount lustre-MDT0001 complete [ 5649.983834] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 20:58:18 (1789088298) [ 5658.032360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5658.636001] LustreError: 193121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5658.734258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5663.328843] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5664.228813] LustreError: 193122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5668.322909] LustreError: 193121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5672.199968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5672.687283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5677.038608] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5685.153913] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5691.218757] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5698.326633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5698.667618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5703.785554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5706.110907] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5715.311506] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5723.046097] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5723.220924] Lustre: lustre-OST0001: new disk, initializing [ 5723.224824] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5723.230423] Lustre: 195963:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5723.300535] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5729.819365] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5729.886423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5729.896563] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5730.023725] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5739.656501] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5743.073681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5743.085494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5743.118646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5748.720680] Lustre: server umount lustre-OST0000 complete [ 5750.547047] LustreError: 196251:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5755.447974] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5755.758994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5760.186782] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5760.433593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5769.186952] LustreError: 196251:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5769.223353] LustreError: 196251:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 5776.352960] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5776.357169] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5776.367299] LustreError: Skipped 2 previous similar messages [ 5776.369087] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5776.389563] Lustre: Skipped 2 previous similar messages [ 5776.418644] Lustre: Skipped 4 previous similar messages [ 5781.803432] Lustre: server umount lustre-OST0001 complete [ 5791.941247] Lustre: server umount lustre-MDT0000 complete [ 5795.648559] LustreError: 193102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789088446 with bad export cookie 256699282677246167 [ 5795.657427] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5795.661565] LustreError: 193102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5796.077462] Lustre: server umount lustre-MDT0001 complete [ 5821.433142] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 21:01:10 (1789088470) [ 5827.639339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5828.021650] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5828.069868] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5828.258614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5832.892480] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5840.663128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5840.783153] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5840.831555] Lustre: 199418:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5846.072242] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5854.165261] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5861.225420] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5868.082350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5868.288025] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5874.839423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5875.276768] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5883.142681] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5889.758292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5889.867780] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5895.918527] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5896.661363] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5905.145597] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5910.498788] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5910.515320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5910.538754] Lustre: Skipped 2 previous similar messages [ 5910.550942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5910.559237] Lustre: Skipped 4 previous similar messages [ 5914.121863] Lustre: server umount lustre-OST0000 complete [ 5915.917157] LustreError: 201268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5915.939177] LustreError: 201268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 5932.022777] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5932.036566] LustreError: Skipped 1 previous similar message [ 5936.819916] Lustre: server umount lustre-OST0001 complete [ 5946.990865] Lustre: server umount lustre-MDT0000 complete [ 5950.350306] LustreError: 202305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789088601 with bad export cookie 256699282677247574 [ 5950.351531] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5950.361027] LustreError: 202305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5950.747516] Lustre: server umount lustre-MDT0001 complete [ 5973.944660] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 21:03:43 (1789088623) [ 5988.716350] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 5997.667331] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 6031.724863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6031.926344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6031.958976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6032.013265] Lustre: lustre-MDT0000: new disk, initializing [ 6032.096210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6032.101538] Lustre: Skipped 3 previous similar messages [ 6032.112185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6037.005168] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6048.731032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6048.948440] Lustre: 207519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6049.021787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6049.032436] Lustre: Skipped 1 previous similar message [ 6049.147147] Lustre: lustre-MDT0001: new disk, initializing [ 6049.308917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6049.326964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6055.432722] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6066.381807] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6072.501815] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6080.235819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6080.605952] Lustre: lustre-OST0000: new disk, initializing [ 6080.609483] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6080.622971] Lustre: 209487:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6087.415301] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6088.796688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6088.822882] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6088.924445] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6097.428126] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6102.201659] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6102.415701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6107.548639] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6107.766875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6114.790654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6114.800912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6114.818970] Lustre: Skipped 5 previous similar messages [ 6114.821425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6114.835646] LustreError: Skipped 1 previous similar message [ 6114.838029] Lustre: Skipped 8 previous similar messages [ 6116.618076] Lustre: server umount lustre-OST0000 complete [ 6127.188651] Lustre: server umount lustre-MDT0000 complete [ 6130.145693] LustreError: 207532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6130.182973] LustreError: 207532:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6131.634724] LustreError: 207512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789088782 with bad export cookie 256699282677249352 [ 6131.647876] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6131.656251] LustreError: 207512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6132.230092] Lustre: server umount lustre-MDT0001 complete [ 6140.758975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6144.950883] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6152.319014] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6156.342507] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6162.533441] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6167.216744] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6173.134305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6173.460291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6173.464857] Lustre: Skipped 4 previous similar messages [ 6179.151695] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6186.268183] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6191.137264] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6191.488161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6196.361432] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6196.624195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6202.343865] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6202.350771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6202.356717] LustreError: Skipped 1 previous similar message [ 6202.375070] Lustre: Skipped 4 previous similar messages [ 6213.087631] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6213.246278] Lustre: server umount lustre-OST0000 complete [ 6219.301872] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6223.903521] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6252.570801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6258.837735] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6265.962819] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6267.309396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.328581] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.334454] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6267.361057] Lustre: Skipped 1 previous similar message [ 6271.698262] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 6277.620948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6277.628333] Lustre: Skipped 9 previous similar messages [ 6281.199132] Lustre: server umount lustre-OST0000 complete [ 6299.104384] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6299.293231] Lustre: server umount lustre-MDT0000 complete [ 6303.129565] LustreError: 211688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789088954 with bad export cookie 256699282677250241 [ 6303.130231] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6303.141339] LustreError: 211688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6303.544852] Lustre: server umount lustre-MDT0001 complete [ 6313.650552] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 21:09:22 (1789088962) [ 6321.694606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6322.188312] LustreError: 217680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6322.201513] LustreError: 217680:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6327.108859] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6335.356116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6340.189986] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6347.324519] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6353.004498] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6359.368524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6365.798839] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6372.151362] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6377.041992] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6377.365470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6381.110781] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6381.271654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6388.867818] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6389.114625] Lustre: lustre-OST0001: new disk, initializing [ 6389.119438] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6389.126039] Lustre: 221049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6395.614179] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6396.982420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6396.996556] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6397.124912] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6405.722634] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6410.549652] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6410.762793] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6415.508200] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6415.749865] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6417.898802] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6417.909935] LustreError: Skipped 3 previous similar messages [ 6417.913224] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6417.923750] Lustre: Skipped 4 previous similar messages [ 6432.223151] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6432.315294] Lustre: server umount lustre-OST0001 complete [ 6438.413381] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6438.667210] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6443.300639] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6443.555196] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6473.929423] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6474.192818] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6474.196931] Lustre: Skipped 5 previous similar messages [ 6479.363233] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6479.379740] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6479.387991] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6480.496737] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6488.539479] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6490.513649] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6499.350577] Lustre: server umount lustre-OST0001 complete [ 6522.849761] Lustre: server umount lustre-OST0000 complete [ 6532.758518] Lustre: server umount lustre-MDT0000 complete [ 6537.102801] LustreError: 217660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789089188 with bad export cookie 256699282677251207 [ 6537.107077] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6537.116102] LustreError: 217660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6543.566723] Lustre: server umount lustre-MDT0001 complete [ 6567.039079] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 21:13:36 (1789089216) [ 6573.820455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6574.089967] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6574.130225] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6578.824245] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6586.980811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6587.103258] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6587.168569] Lustre: 226295:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6591.870857] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6600.412479] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6606.250514] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6613.752526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6613.995753] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6620.495740] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6627.683935] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6636.486851] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6636.551474] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6642.015563] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6649.250434] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6651.158224] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6655.508445] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6655.657295] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6660.724131] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6660.923640] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6667.547702] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8bcfe00c9800.ost_server_uuid 50 [ 6668.957610] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8bcfe00c9800.ost_server_uuid in IDLE state after 0 sec [ 6670.064126] Lustre: Permanently deactivating lustre-OST0001 [ 6670.079319] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6672.920260] Lustre: Permanently reactivating lustre-OST0001 [ 6681.568108] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6681.572800] LustreError: Skipped 6 previous similar messages [ 6681.578029] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6681.592899] Lustre: Skipped 7 previous similar messages [ 6681.602924] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6681.607094] Lustre: Skipped 22 previous similar messages [ 6683.226596] Lustre: server umount lustre-OST0001 complete [ 6687.201269] LustreError: 228142:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6687.229597] LustreError: 228142:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 37 previous similar messages [ 6692.622550] Lustre: server umount lustre-OST0000 complete [ 6710.242080] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6710.513571] Lustre: server umount lustre-MDT0000 complete [ 6713.745618] LustreError: 226289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789089364 with bad export cookie 256699282677252495 [ 6713.748690] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6713.758423] LustreError: 226289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6714.145182] Lustre: server umount lustre-MDT0001 complete [ 6733.588194] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 21:16:23 (1789089383) [ 6740.397188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6740.683333] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6740.706338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6740.713539] Lustre: Skipped 5 previous similar messages [ 6744.813147] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6751.639069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6751.748312] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6751.810140] Lustre: 232990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6756.038266] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6763.226494] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6768.676849] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6775.644903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6775.893334] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6775.917427] Lustre: Permanently deactivating lustre-OST0000 [ 6775.935526] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6781.845910] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6785.573837] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6785.632349] LustreError: 188513:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000eaaa16b x1875994916365440/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6788.848885] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6795.719660] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6795.847197] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6802.418457] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6804.662104] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6809.768503] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6811.426259] Lustre: Permanently reactivating lustre-OST0000 [ 6811.442132] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6811.446180] Lustre: Skipped 2 previous similar messages [ 6812.491850] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6812.498948] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6812.506807] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6812.515580] Lustre: Skipped 1 previous similar message [ 6812.566765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6828.097535] Lustre: server umount lustre-OST0001 complete [ 6837.991361] Lustre: server umount lustre-OST0000 complete [ 6847.582207] Lustre: server umount lustre-MDT0000 complete [ 6851.259242] LustreError: 232982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789089502 with bad export cookie 256699282677254175 [ 6851.267741] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6851.278593] LustreError: 232982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6851.650751] Lustre: server umount lustre-MDT0001 complete [ 6861.146900] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 6863.584364] Key type lgssc unregistered [ 6863.932637] LNet: 237869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6863.942438] LNetError: 237869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6863.959180] LNet: Removed LNI 192.168.206.144@tcp [ 6864.758629] Key type .llcrypt unregistered [ 6864.764837] Key type ._llcrypt unregistered [ 6885.612544] Key type ._llcrypt registered [ 6885.615312] Key type .llcrypt registered [ 6885.703670] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 6886.644800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6886.682586] alg: No test for adler32 (adler32-zlib) [ 6887.778372] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 6888.131171] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 6889.807268] Key type lgssc registered [ 6890.820923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6900.246543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6900.265850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6901.782111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6905.902337] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6911.972376] LustreError: 239339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6911.989216] LustreError: 239339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6913.889667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6914.002030] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6914.032613] Lustre: 239328:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6914.041175] Lustre: Permanently deactivating lustre-MDT0001 [ 6914.052361] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6914.346145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6918.313100] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6919.686840] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6925.613457] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6930.924978] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6937.284946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6937.580828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6939.626596] LustreError: 241179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6939.655062] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6942.951251] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6944.760777] LustreError: 241179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6946.792154] LustreError: 241177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6946.820930] LustreError: 241177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6946.824711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6949.628785] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6951.920732] LustreError: 241319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6951.940358] LustreError: 241319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6955.292862] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6955.466696] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6960.625715] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6960.632827] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6960.824829] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6967.776762] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6971.631987] Lustre: Permanently reactivating lustre-MDT0001 [ 6971.655301] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6971.660241] Lustre: Skipped 2 previous similar messages [ 6973.803164] Lustre: lustre-MDT0001: Client 0979eb58-67b1-4ad3-9717-b2edb1abb058 (at 192.168.206.44@tcp) reconnecting [ 6978.334116] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6978.664770] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6978.678651] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6978.690154] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6978.703329] LustreError: 243096:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 6978.706865] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6978.711393] LustreError: 243096:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 6978.711528] LustreError: 243096:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.144@tcp: cfg command failed: rc = -108 [ 6978.746643] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 6978.746643] [ 6978.751816] LustreError: 239330:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6979.515538] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6984.261312] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6984.422718] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6989.328830] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6989.535972] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6992.388514] Lustre: Permanently deactivating lustre-MDT0001 [ 6992.404456] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6992.409880] Lustre: Skipped 2 previous similar messages [ 7000.144610] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7006.708605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7006.718906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7006.738968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7009.763872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7009.773922] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7009.808569] Lustre: Skipped 1 previous similar message [ 7013.086638] Lustre: server umount lustre-MDT0000 complete [ 7014.889021] LustreError: 242840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7014.914763] LustreError: 242840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7016.615626] LustreError: 239319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789089667 with bad export cookie 4078734425157371243 [ 7016.623315] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7016.627955] LustreError: 239319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7016.876774] Lustre: server umount lustre-MDT0001 complete [ 7022.699457] Lustre: server umount lustre-OST0000 complete [ 7034.044477] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 21:21:23 (1789089683) [ 7043.488852] Lustre: server umount lustre-OST0001 complete [ 7049.301402] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 7057.945734] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 7091.982319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7092.203395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7092.219267] Lustre: Skipped 2 previous similar messages [ 7092.272939] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7092.357449] Lustre: lustre-MDT0000: new disk, initializing [ 7092.440642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7092.451346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7097.073586] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7106.584984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7106.689178] Lustre: 247594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7106.741859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7106.744510] Lustre: Skipped 1 previous similar message [ 7106.796184] Lustre: lustre-MDT0001: new disk, initializing [ 7106.893953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7106.929290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7106.940702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7111.613963] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7120.768189] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7126.577810] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7132.847332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7133.109564] Lustre: lustre-OST0000: new disk, initializing [ 7133.116891] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7133.126339] Lustre: 249562:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7133.236057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7138.576610] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7140.397423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7140.421601] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7140.524777] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7147.522399] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7149.636083] LustreError: 247599:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7157.275382] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7157.424141] Lustre: lustre-OST0001: new disk, initializing [ 7157.435099] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7157.443785] Lustre: 250678:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7159.679162] LustreError: 247599:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7159.702607] LustreError: 247600:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7163.247756] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7166.044910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7166.053838] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7166.167920] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7169.711499] LustreError: 247600:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7171.479218] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7172.959212] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7176.498199] Lustre: lustre-OST0001: Not available for connect from 192.168.206.44@tcp (stopping) [ 7176.514073] Lustre: Skipped 3 previous similar messages [ 7176.704456] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7176.727719] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7176.741630] Lustre: Skipped 1 previous similar message [ 7181.334389] Lustre: server umount lustre-OST0001 complete [ 7181.582217] LustreError: 249548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7202.281557] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7202.283514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7202.292418] LustreError: Skipped 1 previous similar message [ 7202.302132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7202.306529] Lustre: Skipped 2 previous similar messages [ 7202.327332] Lustre: Skipped 3 previous similar messages [ 7203.688142] Lustre: server umount lustre-OST0000 complete [ 7207.395901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7213.381454] Lustre: server umount lustre-MDT0000 complete [ 7216.647093] LustreError: 247586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789089867 with bad export cookie 4078734425157374057 [ 7216.650472] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7216.663515] LustreError: 247586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7223.141657] Lustre: server umount lustre-MDT0001 complete [ 7232.567775] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 7234.876825] Key type lgssc unregistered [ 7235.148492] LNet: 252709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7235.155733] LNetError: 252709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7235.190197] LNet: Removed LNI 192.168.206.144@tcp [ 7235.942624] Key type .llcrypt unregistered [ 7235.945513] Key type ._llcrypt unregistered [ 7270.410891] Key type ._llcrypt registered [ 7270.415731] Key type .llcrypt registered [ 7270.502437] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 7271.589505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7271.615237] alg: No test for adler32 (adler32-zlib) [ 7272.591797] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 7272.786578] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 7274.434267] Key type lgssc registered [ 7275.423450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7281.865746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7281.886682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7283.209792] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7283.253913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7283.444669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7288.121329] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7296.748419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7296.863545] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7296.919382] Lustre: 254915:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7297.275740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7303.246756] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7312.218908] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7317.786908] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7324.670032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7324.981969] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7325.238631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7330.962830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7331.794290] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7338.328317] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7387.104362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7387.113975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7387.128382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7387.630497] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7387.634244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7387.653540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7392.113263] Lustre: server umount lustre-OST0000 complete [ 7395.652263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7406.819769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7407.116204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7408.189665] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7408.210619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7408.232221] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7408.243633] Lustre: Skipped 1 previous similar message [ 7412.953572] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7419.816223] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7459.808695] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7459.809110] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7459.819715] LustreError: Skipped 1 previous similar message [ 7459.834753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7459.852771] Lustre: Skipped 1 previous similar message [ 7464.419305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7464.436330] Lustre: Skipped 1 previous similar message [ 7466.144596] Lustre: server umount lustre-OST0000 complete [ 7471.584438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7471.592222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7471.604578] Lustre: Skipped 1 previous similar message [ 7471.612371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7476.604618] Lustre: server umount lustre-MDT0000 complete [ 7480.596126] LustreError: 254909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789090131 with bad export cookie 2492217607949222334 [ 7480.610462] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7481.057668] Lustre: server umount lustre-MDT0001 complete [ 7491.977297] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 7494.858874] Key type lgssc unregistered [ 7495.254957] LNet: 260338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7495.260597] LNetError: 260338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7496.300732] LNet: Removed LNI 192.168.206.144@tcp [ 7497.688168] Key type .llcrypt unregistered [ 7497.694786] Key type ._llcrypt unregistered [ 7526.460923] Key type ._llcrypt registered [ 7526.462481] Key type .llcrypt registered [ 7526.662254] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 7527.734919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7527.776457] alg: No test for adler32 (adler32-zlib) [ 7528.782852] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 7529.126794] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 7530.785254] Key type lgssc registered [ 7531.811508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7538.904807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7538.936087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7540.580873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7544.572516] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7550.945960] LustreError: 261755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7550.981912] LustreError: 261755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7552.336638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7552.660846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7557.179501] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7564.007869] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7569.120176] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7575.347593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7575.674404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7581.129797] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7585.920659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7588.288346] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7608.800149] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7608.804764] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7608.812464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7611.874773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7611.881530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7611.881757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7614.953838] Lustre: server umount lustre-OST0000 complete [ 7619.554366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7619.562243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7619.570050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7619.572571] Lustre: Skipped 1 previous similar message [ 7622.113108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7622.113636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7624.564779] Lustre: server umount lustre-MDT0000 complete [ 7627.238148] LustreError: 261756:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7627.264171] LustreError: 261756:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7628.732817] LustreError: 261736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789090279 with bad export cookie 2188366351685931781 [ 7628.763794] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7629.055747] Lustre: server umount lustre-MDT0001 complete [ 7639.203492] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 7642.207177] Key type lgssc unregistered [ 7642.525611] LNet: 266058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7642.542184] LNetError: 266058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7642.569579] LNet: Removed LNI 192.168.206.144@tcp [ 7643.413549] Key type .llcrypt unregistered [ 7643.416244] Key type ._llcrypt unregistered [ 7662.566751] Key type ._llcrypt registered [ 7662.575673] Key type .llcrypt registered [ 7662.707242] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 7663.972102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7664.210161] alg: No test for adler32 (adler32-zlib) [ 7665.623505] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 7666.085498] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 7667.751211] Key type lgssc registered [ 7669.631139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7679.384988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7679.410501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7680.930501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7685.398165] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7690.226230] LustreError: 267280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7690.238275] LustreError: 267280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7693.640613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7694.108531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7698.031820] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7705.618774] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7711.662550] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7718.189727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7718.527450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7723.777647] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7725.742568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7730.306497] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7741.408418] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7741.416308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7741.422219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7741.436959] Lustre: Skipped 1 previous similar message [ 7743.189707] Lustre: server umount lustre-OST0000 complete [ 7750.626319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7750.629071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7750.638684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7750.643179] Lustre: Skipped 1 previous similar message [ 7751.648196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7751.662441] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7753.754830] Lustre: server umount lustre-MDT0000 complete [ 7756.769447] LustreError: 267276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7756.790302] LustreError: 267276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7757.802851] LustreError: 269918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789090408 with bad export cookie 14561982142518056214 [ 7757.806396] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7757.813399] LustreError: 269918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7758.195312] Lustre: server umount lustre-MDT0001 complete [ 7769.513256] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 7771.977271] Key type lgssc unregistered [ 7772.208587] LNet: 270957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7772.214582] LNetError: 270957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7772.231639] LNet: Removed LNI 192.168.206.144@tcp [ 7772.881726] Key type .llcrypt unregistered [ 7772.884119] Key type ._llcrypt unregistered [ 7793.081580] Key type ._llcrypt registered [ 7793.083868] Key type .llcrypt registered [ 7793.168111] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 7793.955120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7793.977251] alg: No test for adler32 (adler32-zlib) [ 7794.942727] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 7795.157289] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 7796.799166] Key type lgssc registered [ 7797.608742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7803.312895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7803.333538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7804.777512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7808.343919] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7814.114398] LustreError: 272353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7814.133130] LustreError: 272353:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7815.140553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7815.433373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7818.514133] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7823.805664] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7827.860275] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7833.278082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7833.657241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7839.291542] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7847.348754] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7850.027886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7875.553753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7875.569842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7875.585121] Lustre: Skipped 1 previous similar message [ 7875.602494] Lustre: Skipped 1 previous similar message [ 7878.772951] Lustre: server umount lustre-OST0000 complete [ 7885.793051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7885.812169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7885.815721] Lustre: Skipped 1 previous similar message [ 7885.833527] Lustre: Skipped 1 previous similar message [ 7888.778746] Lustre: server umount lustre-MDT0000 complete [ 7890.912967] LustreError: 273080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7890.928310] LustreError: 273080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7892.658178] LustreError: 272334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789090543 with bad export cookie 4562101351631412592 [ 7892.664231] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7892.667816] LustreError: 272334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7892.883570] Lustre: server umount lustre-MDT0001 complete [ 7902.699279] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 7905.255359] Key type lgssc unregistered [ 7905.503038] LNet: 276658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7905.516851] LNetError: 276658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7905.534806] LNet: Removed LNI 192.168.206.144@tcp [ 7906.369337] Key type .llcrypt unregistered [ 7906.373575] Key type ._llcrypt unregistered [ 7924.628393] Key type ._llcrypt registered [ 7924.630828] Key type .llcrypt registered [ 7924.725563] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 7925.735464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7925.748267] alg: No test for adler32 (adler32-zlib) [ 7926.919961] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 7927.213501] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 7928.937102] Key type lgssc registered [ 7929.851481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7936.829847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7936.842886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7938.135875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7941.311297] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7947.924688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7948.291423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7951.799207] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7957.622172] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7962.750927] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7967.788174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7968.065736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7972.919199] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7979.309499] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7984.434225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7993.824695] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7993.830853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7993.842291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7994.850057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7994.852253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7994.859586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7998.806140] Lustre: server umount lustre-OST0000 complete [ 8004.583968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8004.589397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8004.598903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8004.604353] Lustre: Skipped 1 previous similar message [ 8007.985578] Lustre: server umount lustre-MDT0000 complete [ 8010.211050] LustreError: 277906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8010.222566] LustreError: 277906:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8010.712552] LustreError: 277860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789090661 with bad export cookie 9785320071986602316 [ 8010.719601] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8010.721879] LustreError: 277860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8010.950282] Lustre: server umount lustre-MDT0001 complete [ 8019.105369] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 8021.414405] Key type lgssc unregistered [ 8021.667734] LNet: 281593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8021.673458] LNetError: 281593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8021.688882] LNet: Removed LNI 192.168.206.144@tcp [ 8022.354351] Key type .llcrypt unregistered [ 8022.359318] Key type ._llcrypt unregistered [ 8032.917693] Key type ._llcrypt registered [ 8032.919481] Key type .llcrypt registered [ 8033.005450] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8049.296595] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 8063.043915] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 8063.574794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8063.587758] alg: No test for adler32 (adler32-zlib) [ 8064.679540] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 8064.901570] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 8066.599166] Key type lgssc registered [ 8067.572409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8099.780182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8099.810880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8101.064462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8101.093146] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8101.171225] Lustre: lustre-MDT0000: new disk, initializing [ 8101.246198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8101.271455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8104.769507] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8113.357350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8113.499434] Lustre: 285141:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8113.586153] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8113.599087] Lustre: Skipped 1 previous similar message [ 8113.711632] Lustre: lustre-MDT0001: new disk, initializing [ 8113.879915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8113.915838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8113.934516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8118.143739] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8126.801133] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8132.267287] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8138.099199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8138.348332] Lustre: lustre-OST0000: new disk, initializing [ 8138.354929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8138.363211] Lustre: 287109:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8138.462758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8143.792329] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8147.554247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8147.562056] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8147.646275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8153.593813] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8158.474201] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8158.645374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8164.068507] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8164.352630] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8168.429407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8168.445400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8168.450757] Lustre: Skipped 1 previous similar message [ 8168.453303] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8168.476283] Lustre: Skipped 1 previous similar message [ 8172.534767] Lustre: server umount lustre-OST0000 complete [ 8178.660036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8178.694796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8178.699973] Lustre: Skipped 1 previous similar message [ 8182.879685] Lustre: server umount lustre-MDT0000 complete [ 8185.773908] LustreError: 285133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789090836 with bad export cookie 8074499312885240758 [ 8185.785863] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8186.028884] Lustre: server umount lustre-MDT0001 complete [ 8193.378777] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 21:40:43 (1789090843) [ 8199.249867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8199.648361] LustreError: 289522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8199.663394] LustreError: 289522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8199.742335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8204.091309] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8204.771740] LustreError: 289523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8209.895107] LustreError: 289522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8213.502450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8213.845552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8218.609755] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8227.101085] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8232.258973] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8238.397461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8238.501690] LustreError: 291341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8238.520383] LustreError: 291341:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8238.660410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8243.624625] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8250.377644] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8259.044210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8259.055679] LustreError: Skipped 1 previous similar message [ 8259.060804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8259.067614] Lustre: Skipped 1 previous similar message [ 8259.082117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8261.090291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8261.093099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8261.107837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8265.004935] Lustre: server umount lustre-OST0000 complete [ 8270.304446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8270.317246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8270.327698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8270.336123] Lustre: Skipped 1 previous similar message [ 8274.604656] Lustre: server umount lustre-MDT0000 complete [ 8276.461459] LustreError: 290251:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8276.481981] LustreError: 290251:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8277.934848] LustreError: 289503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789090928 with bad export cookie 8074499312885241640 [ 8277.940301] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8277.945560] LustreError: 289503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8278.226863] Lustre: server umount lustre-MDT0001 complete [ 8287.415330] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 8289.640667] Key type lgssc unregistered [ 8289.900378] LNet: 293056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8289.909387] LNetError: 293056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8289.925698] LNet: Removed LNI 192.168.206.144@tcp [ 8290.579777] Key type .llcrypt unregistered [ 8290.581838] Key type ._llcrypt unregistered [ 8311.385986] Key type ._llcrypt registered [ 8311.388242] Key type .llcrypt registered [ 8311.483628] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 8323.352793] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 8324.131848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8324.177274] alg: No test for adler32 (adler32-zlib) [ 8325.213715] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 8325.412780] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 8327.080433] Key type lgssc registered [ 8328.145652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8367.978180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8368.011279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8369.240325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8369.268610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8369.361463] Lustre: lustre-MDT0000: new disk, initializing [ 8369.415242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8369.429322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8373.276499] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8382.361263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8382.442673] Lustre: 296859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8382.496623] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8382.500915] Lustre: Skipped 1 previous similar message [ 8382.573927] Lustre: lustre-MDT0001: new disk, initializing [ 8382.731113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8382.778643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8382.788817] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8386.122743] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8394.823721] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8399.929541] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8405.438926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8405.769610] Lustre: lustre-OST2710: new disk, initializing [ 8405.773836] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8405.779641] Lustre: 298829:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8405.866592] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8413.681357] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8414.780566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8414.788833] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8414.879058] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8422.956813] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8429.925816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8430.121649] Lustre: lustre-OST03e8: new disk, initializing [ 8430.125238] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8430.132384] Lustre: 299894:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8430.218391] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8435.901848] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8436.822510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8436.836735] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8436.915411] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8442.463600] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8449.857781] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8450.136084] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8453.686654] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8453.952312] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8458.643793] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8458.834442] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8463.877025] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8464.145376] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8467.937607] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8467.942192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8467.977613] Lustre: Skipped 1 previous similar message [ 8467.998396] Lustre: Skipped 3 previous similar messages [ 8473.062616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8473.065683] Lustre: Skipped 2 previous similar messages [ 8474.216737] Lustre: server umount lustre-MDT0000 complete [ 8477.507219] LustreError: 296854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789091128 with bad export cookie 3465377530035165006 [ 8477.513269] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8477.523489] LustreError: 296854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8477.918477] Lustre: server umount lustre-MDT0001 complete [ 8492.521815] Lustre: server umount lustre-OST2710 complete [ 8494.367220] Lustre: 294659:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789091129/real 1789091129] req@00000000bb777afe x1875997859315584/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789091145 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8494.396771] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8494.412429] Lustre: Skipped 2 previous similar messages [ 8496.201885] Lustre: server umount lustre-OST03e8 complete [ 8513.623370] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 8524.935500] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 8562.762033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8563.059531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8563.080492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8563.178798] Lustre: lustre-MDT0000: new disk, initializing [ 8563.265354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8563.284394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8567.438399] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8579.531898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8579.641653] Lustre: 305145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8579.690370] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8579.694398] Lustre: Skipped 1 previous similar message [ 8579.797372] Lustre: lustre-MDT0001: new disk, initializing [ 8579.892466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8579.934652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8579.958655] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8584.215021] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8594.319291] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8599.547664] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8605.651156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8605.955256] Lustre: lustre-OST0000: new disk, initializing [ 8605.959569] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8605.975281] Lustre: 307114:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8606.111800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8612.869232] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8612.901383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8612.911801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8613.036481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8622.012267] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8625.562241] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8625.753653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8629.599297] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8629.795692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8633.322835] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8633.336127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8633.362180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8633.824901] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8637.484977] Lustre: server umount lustre-OST0000 complete [ 8641.505625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8641.512427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8641.524829] Lustre: Skipped 1 previous similar message [ 8641.529848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8641.533986] Lustre: Skipped 1 previous similar message [ 8646.645165] Lustre: server umount lustre-MDT0000 complete [ 8649.491473] LustreError: 305137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789091300 with bad export cookie 3465377530035166686 [ 8649.492484] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8649.497470] LustreError: 305137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8649.748870] Lustre: server umount lustre-MDT0001 complete [ 8656.296792] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 21:48:26 (1789091306) [ 8657.694433] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8659.162157] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 21:48:28 (1789091308) [ 8662.709761] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 8686.933451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8687.130834] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8687.152951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8687.260849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8691.713504] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8701.443289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8701.560854] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8701.616509] Lustre: 310970:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8706.783365] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8728.689154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8728.992870] Lustre: Denying initial registration attempt from nid 192.168.206.144@tcp, specified as failover [ 8729.013938] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8729.025580] LustreError: 312425:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8729.059742] LustreError: 312425:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8729.079475] LustreError: 312425:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8729.204902] Lustre: server umount lustre-OST0000 complete [ 8729.223867] LustreError: 312425:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8737.760579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8737.766268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8737.773156] Lustre: Skipped 1 previous similar message [ 8737.779981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8737.787254] Lustre: Skipped 2 previous similar messages [ 8749.026133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8749.164780] Lustre: server umount lustre-MDT0000 complete [ 8749.539120] LustreError: 310982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8749.565311] LustreError: 310982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8752.083942] LustreError: 310962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789091402 with bad export cookie 3465377530035167568 [ 8752.088432] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8752.094462] LustreError: 310962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8752.324710] Lustre: server umount lustre-MDT0001 complete [ 8761.787285] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 8764.067784] Key type lgssc unregistered [ 8764.359267] LNet: 313546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8764.366789] LNetError: 313546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8764.378944] LNet: Removed LNI 192.168.206.144@tcp [ 8765.126854] Key type .llcrypt unregistered [ 8765.128958] Key type ._llcrypt unregistered [ 8776.613509] Key type ._llcrypt registered [ 8776.615948] Key type .llcrypt registered [ 8776.748796] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 8777.550229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8777.597779] alg: No test for adler32 (adler32-zlib) [ 8778.790343] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 8779.064924] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 8780.753056] Key type lgssc registered [ 8782.057701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8823.856863] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 8830.909629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8830.939843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8832.189949] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8832.203381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8832.314802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8836.392463] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8844.301545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8844.421178] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8844.447600] Lustre: 316292:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8844.735866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8848.598544] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8854.074378] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8858.680779] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8876.522862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8876.806251] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8876.998586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8882.465615] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8889.061220] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8894.951604] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8894.963182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8894.981052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8895.456194] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8895.466123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8900.577382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8900.588783] Lustre: Skipped 3 previous similar messages [ 8905.183524] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8905.285644] Lustre: server umount lustre-OST0000 complete [ 8910.825566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8910.827494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8910.845540] Lustre: Skipped 1 previous similar message [ 8910.854215] Lustre: Skipped 1 previous similar message [ 8915.014543] Lustre: server umount lustre-MDT0000 complete [ 8918.497448] LustreError: 317031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789091569 with bad export cookie 1284715529849866537 [ 8918.500938] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8918.506923] LustreError: 317031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8918.731379] Lustre: server umount lustre-MDT0001 complete [ 8926.036494] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 8928.529899] Key type lgssc unregistered [ 8928.756647] LNet: 319861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8928.765519] LNetError: 319861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8929.830713] LNet: Removed LNI 192.168.206.144@tcp [ 8930.528122] Key type .llcrypt unregistered [ 8930.529627] Key type ._llcrypt unregistered [ 8952.633382] Key type ._llcrypt registered [ 8952.634951] Key type .llcrypt registered [ 8952.764522] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 8953.598742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8953.608542] alg: No test for adler32 (adler32-zlib) [ 8954.594302] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 8954.798527] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 8956.487199] Key type lgssc registered [ 8957.515728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8964.629300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8964.659659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8966.140028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8969.926388] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8976.354174] LustreError: 321280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8976.373047] LustreError: 321280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8977.344781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8977.649467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8981.383862] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8987.441630] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8991.689954] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8998.826634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8999.097121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9003.473310] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9009.782480] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9023.971478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9023.982100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9023.997122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9026.839165] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.44@tcp (stopping) [ 9027.043737] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9028.674387] Lustre: server umount lustre-MDT0000 complete [ 9031.055444] LustreError: 321258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789091681 with bad export cookie 1474933419113193483 [ 9031.062171] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9031.063265] LustreError: 321258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9031.945156] LustreError: 321276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9031.972213] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.44@tcp (stopping) [ 9031.974707] Lustre: Skipped 3 previous similar messages [ 9032.162139] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9032.171189] Lustre: Skipped 1 previous similar message [ 9036.192599] LustreError: 321276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9036.192709] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9036.216270] LustreError: 321276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9036.227262] Lustre: Skipped 1 previous similar message [ 9037.360895] Lustre: server umount lustre-MDT0001 complete [ 9041.945498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9048.532618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9055.914700] LustreError: 323109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9055.930056] LustreError: 323109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9055.968533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9056.021576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9059.582138] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9061.093189] LustreError: 324893:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9061.109410] LustreError: 324893:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9065.191226] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9067.234574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9067.594364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9070.802684] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9072.617660] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9077.547315] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9078.996590] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9084.283968] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9086.107776] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9093.092308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9093.093394] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9093.098816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9093.098824] Lustre: Skipped 1 previous similar message [ 9093.108937] Lustre: Skipped 1 previous similar message [ 9098.878514] Lustre: server umount lustre-OST0000 complete [ 9103.329452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9103.331421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9103.347111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9103.351923] Lustre: Skipped 4 previous similar messages [ 9108.448725] LustreError: 324891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9108.468568] LustreError: 324891:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9108.527466] Lustre: server umount lustre-MDT0000 complete [ 9112.059759] LustreError: 321260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789091762 with bad export cookie 1474933419113201316 [ 9112.068659] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9112.073954] LustreError: 321260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9112.389697] Lustre: server umount lustre-MDT0001 complete [ 9122.459144] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 9125.512187] Key type lgssc unregistered [ 9125.796608] LNet: 327565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9125.808143] LNetError: 327565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9125.838280] LNet: Removed LNI 192.168.206.144@tcp [ 9127.029900] Key type .llcrypt unregistered [ 9127.035751] Key type ._llcrypt unregistered [ 9152.467813] Key type ._llcrypt registered [ 9152.475716] Key type .llcrypt registered [ 9152.657549] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 9154.037284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9154.257853] alg: No test for adler32 (adler32-zlib) [ 9155.602458] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 9156.003826] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 9157.743665] Key type lgssc registered [ 9159.277059] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9166.633720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9166.652423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9167.980337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9172.157904] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9177.584046] LustreError: 328980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9177.594907] LustreError: 328980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9180.075455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9180.361737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9183.677669] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9190.767165] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9197.209754] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9201.122397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9201.126612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9201.134475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9203.175423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9203.182993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9205.283296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9205.301775] Lustre: Skipped 2 previous similar messages [ 9207.245118] Lustre: server umount lustre-MDT0000 complete [ 9210.346738] LustreError: 328980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9210.399948] LustreError: 328980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9211.468958] LustreError: 329713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789091862 with bad export cookie 15190480164516812092 [ 9211.474561] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9211.485676] LustreError: 329713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9212.041771] Lustre: server umount lustre-MDT0001 complete [ 9224.611146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9224.927868] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9224.954195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9225.171187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9229.176236] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9237.412326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9237.531520] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9237.576576] Lustre: 331566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9237.702613] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9237.898583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9241.779990] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9250.133591] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9255.949405] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9266.942946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9267.330947] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9267.341944] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9267.355658] LustreError: 333450:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9267.361278] LustreError: 333450:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9267.368844] LustreError: 333450:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9267.484348] Lustre: server umount lustre-OST0000 complete [ 9267.488881] LustreError: 333450:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9281.128929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9281.482259] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9281.733166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9288.695487] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9289.394730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9297.085911] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9305.752246] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9305.843992] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9305.921783] Lustre: lustre-OST0001: new disk, initializing [ 9305.924837] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9305.930332] Lustre: 334997:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9305.993853] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9313.082851] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9313.991387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9314.001205] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9314.165668] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9324.729219] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9329.645535] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9329.649803] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9329.675206] LustreError: Skipped 1 previous similar message [ 9329.690022] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9329.713367] Lustre: Skipped 1 previous similar message [ 9335.566595] Lustre: server umount lustre-OST0001 complete [ 9339.891203] LustreError: 333918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9339.920143] LustreError: 333918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9340.901395] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9340.911732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9340.922270] Lustre: Skipped 1 previous similar message [ 9340.926762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9340.936188] Lustre: Skipped 2 previous similar messages [ 9345.000229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9345.003127] LustreError: 333918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9345.043478] LustreError: 333918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9350.124570] LustreError: 334320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9350.141245] LustreError: 334320:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9355.236427] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9355.472348] Lustre: server umount lustre-OST0000 complete [ 9360.352377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9360.359548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9360.384348] Lustre: Skipped 1 previous similar message [ 9360.388295] Lustre: Skipped 6 previous similar messages [ 9365.411618] Lustre: server umount lustre-MDT0000 complete [ 9365.472984] LustreError: 331573:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9365.485335] LustreError: 331573:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9368.955547] LustreError: 331557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789092019 with bad export cookie 15190480164516812645 [ 9368.960317] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9368.966166] LustreError: 331557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9369.265887] Lustre: server umount lustre-MDT0001 complete [ 9379.947552] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 9383.060981] Key type lgssc unregistered [ 9383.407778] LNet: 336958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9383.416385] LNetError: 336958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9383.435701] LNet: Removed LNI 192.168.206.144@tcp [ 9384.144889] Key type .llcrypt unregistered [ 9384.147352] Key type ._llcrypt unregistered [ 9460.840244] Key type ._llcrypt registered [ 9460.843570] Key type .llcrypt registered [ 9460.953769] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 9474.460665] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 9475.589029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9475.600810] alg: No test for adler32 (adler32-zlib) [ 9476.630546] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 9476.861978] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 9478.591166] Key type lgssc registered [ 9479.536106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9512.003531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9512.070595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9513.416743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9513.436692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9513.557261] Lustre: lustre-MDT0000: new disk, initializing [ 9513.733307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9513.773870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9519.526721] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9529.568700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9529.670379] Lustre: 341894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9529.725208] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9529.728504] Lustre: Skipped 1 previous similar message [ 9529.808387] Lustre: lustre-MDT0001: new disk, initializing [ 9529.874624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9529.906101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9529.915685] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9534.257281] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9543.166824] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9549.040339] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9555.430606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9555.827260] Lustre: lustre-OST0000: new disk, initializing [ 9555.832213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9555.839662] Lustre: 343862:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9555.959495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9561.159858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9561.167395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9561.310714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9561.525642] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9570.487225] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9575.943347] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9576.114110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9583.327704] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9583.612713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9587.174143] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9587.180578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9587.190915] LustreError: Skipped 1 previous similar message [ 9587.199490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9587.202261] Lustre: Skipped 1 previous similar message [ 9587.228737] Lustre: Skipped 1 previous similar message [ 9592.306277] Lustre: server umount lustre-OST0000 complete [ 9601.511957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9601.535764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9601.559678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9604.069344] Lustre: server umount lustre-MDT0000 complete [ 9608.174755] LustreError: 341884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789092259 with bad export cookie 7177463314265751315 [ 9608.202729] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9608.222042] LustreError: 341884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9608.761609] Lustre: server umount lustre-MDT0001 complete [ 9618.440989] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 22:04:28 (1789092268) [ 9628.251164] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 22:04:38 (1789092278) [ 9639.324740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9639.911496] LustreError: 346621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9639.948246] LustreError: 346621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9640.075878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9644.465768] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9645.537641] LustreError: 346622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9645.558893] LustreError: 346622:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9649.642796] LustreError: 346621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9651.678257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9651.977345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9655.770382] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9662.084867] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9667.791532] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9673.873463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9674.211716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9678.953269] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9685.206115] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9692.055520] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9693.578549] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9695.057898] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9696.760256] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9698.713941] Lustre: Failing over lustre-MDT0000 [ 9699.055928] Lustre: server umount lustre-MDT0000 complete [ 9699.808541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9699.813052] LustreError: 347348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9699.831187] Lustre: Skipped 3 previous similar messages [ 9699.860751] LustreError: 347348:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9704.932461] LustreError: 346621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9704.952736] LustreError: 346621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9715.169015] Lustre: 340120:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789092350/real 1789092350] req@000000005b446e67 x1875999066230144/t0(0) o400->MGC192.168.206.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789092366 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9715.171214] LustreError: 347348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9715.204466] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9715.225945] LustreError: 347348:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 9718.103460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9725.417068] LustreError: 340117:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fd950269 x1875999066235648/t0(0) o250->MGC192.168.206.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9725.741249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9725.793401] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9727.791036] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9731.061946] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9731.101884] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9731.185417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9731.832964] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9742.510953] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9744.419762] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9748.465034] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9750.265980] Lustre: Failing over lustre-MDT0000 [ 9750.710765] Lustre: server umount lustre-MDT0000 complete [ 9751.525770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9751.529620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9751.533878] LustreError: 347715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9751.534398] Lustre: Skipped 1 previous similar message [ 9751.556746] LustreError: 347715:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 9767.913148] Lustre: 340120:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789092402/real 1789092402] req@0000000086cdd2cb x1875999066251008/t0(0) o400->MGC192.168.206.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789092418 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9767.942929] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9769.184178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9777.367714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9777.417252] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9779.978565] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9781.500202] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9782.772180] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9782.776920] Lustre: Skipped 2 previous similar messages [ 9782.808721] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9782.844393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9788.739904] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9789.973592] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9798.114185] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9798.117535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9798.135880] Lustre: Skipped 2 previous similar messages [ 9798.145898] Lustre: Skipped 4 previous similar messages [ 9801.752721] Lustre: server umount lustre-MDT0000 complete [ 9803.237036] LustreError: 347348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9803.264905] LustreError: 347348:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [ 9804.834201] LustreError: 346603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789092455 with bad export cookie 7177463314265754059 [ 9804.841575] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9805.064887] Lustre: server umount lustre-MDT0001 complete [ 9819.138839] Lustre: server umount lustre-OST0000 complete [ 9833.792119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9838.689895] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9849.532540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9849.953818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9849.961996] Lustre: Skipped 1 previous similar message [ 9855.212195] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9866.488438] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9873.623381] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9881.028455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9887.763437] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9895.307572] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9905.012163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9910.244496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9910.247514] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9910.252436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9910.262622] Lustre: Skipped 2 previous similar messages [ 9912.243935] Lustre: server umount lustre-OST0000 complete [ 9920.491205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9920.512039] Lustre: Skipped 2 previous similar messages [ 9922.709327] Lustre: server umount lustre-MDT0000 complete [ 9925.605254] LustreError: 352955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9925.657704] LustreError: 352955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9928.515300] LustreError: 352940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789092579 with bad export cookie 7177463314265754773 [ 9928.534165] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9928.551936] LustreError: 352940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9929.024271] Lustre: server umount lustre-MDT0001 complete [ 9940.405288] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 9943.611765] Key type lgssc unregistered [ 9943.872609] LNet: 356516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9943.883379] LNetError: 356516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9943.903449] LNet: Removed LNI 192.168.206.144@tcp [ 9944.978147] Key type .llcrypt unregistered [ 9944.982865] Key type ._llcrypt unregistered [ 9968.293524] Key type ._llcrypt registered [ 9968.295758] Key type .llcrypt registered [ 9968.368087] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 9969.097290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9969.152954] alg: No test for adler32 (adler32-zlib) [ 9970.183972] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 9970.342555] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 9971.968968] Key type lgssc registered [ 9973.217697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9980.426088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9980.461400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9981.857311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9985.591582] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9991.141512] LustreError: 358034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9991.160232] LustreError: 358034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9993.848793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9994.220031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9998.288212] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10006.443696] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10012.124915] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10018.582644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10018.985657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10025.304675] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10026.185307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10032.590735] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10045.093840] Lustre: server umount lustre-MDT0000 complete [10045.409644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10045.424367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10045.457762] LustreError: 358763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10046.959416] LustreError: 358034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10046.968446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10049.172664] LustreError: 358770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789092700 with bad export cookie 7504490181921571377 [10049.173865] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10049.181442] LustreError: 358770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10049.408595] Lustre: server umount lustre-MDT0001 complete [10064.335484] Lustre: server umount lustre-OST0000 complete [10077.956204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10078.391735] LustreError: 361746:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10078.414114] LustreError: 361746:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10078.503204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10082.945494] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10083.811735] LustreError: 361747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10091.629939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10091.993013] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10096.014382] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10102.821441] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10107.852756] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10114.128700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10114.480562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10119.892802] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10123.706444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10128.145224] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10139.113286] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10139.116468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10139.126281] Lustre: Skipped 2 previous similar messages [10139.143357] Lustre: Skipped 2 previous similar messages [10144.230843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10144.251239] Lustre: Skipped 2 previous similar messages [10145.103037] Lustre: server umount lustre-MDT0000 complete [10148.105687] LustreError: 361727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789092799 with bad export cookie 7504490181921572497 [10148.106518] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10148.123922] LustreError: 361727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10148.417730] Lustre: server umount lustre-MDT0001 complete [10161.765777] Lustre: server umount lustre-OST0000 complete [10174.515434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10174.869818] LustreError: 365654:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10174.888875] LustreError: 365654:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10174.937037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10178.166568] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10184.789674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10188.794170] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10195.529915] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10199.738668] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10204.910045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10210.028993] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10216.248476] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10219.566770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10229.738365] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10229.739745] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10229.753090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10229.767081] LustreError: Skipped 1 previous similar message [10229.793133] Lustre: Skipped 2 previous similar messages [10234.853813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10234.862036] Lustre: Skipped 1 previous similar message [10239.974637] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10239.982075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10240.004842] Lustre: Skipped 2 previous similar messages [10240.308610] Lustre: server umount lustre-OST0000 complete [10246.623996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10246.634662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10250.210252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10250.214288] Lustre: Skipped 1 previous similar message [10252.012232] Lustre: server umount lustre-MDT0000 complete [10255.137476] LustreError: 365634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789092906 with bad export cookie 7504490181921573519 [10255.141078] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10255.146717] LustreError: 365634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10255.478280] Lustre: server umount lustre-MDT0001 complete [10266.454805] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [10268.761957] Key type lgssc unregistered [10269.014893] LNet: 369188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10269.021111] LNetError: 369188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10269.035470] LNet: Removed LNI 192.168.206.144@tcp [10269.965746] Key type .llcrypt unregistered [10269.969829] Key type ._llcrypt unregistered [10297.799727] Key type ._llcrypt registered [10297.804111] Key type .llcrypt registered [10297.925170] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [10299.292360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10299.301374] alg: No test for adler32 (adler32-zlib) [10300.327678] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [10300.609531] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [10302.290749] Key type lgssc registered [10303.530789] Lustre: Echo OBD driver; http://www.lustre.org/ [10310.367029] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10310.383753] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10310.390025] LustreError: 370632:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10310.415984] LustreError: 370632:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10310.421477] LustreError: 370632:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10310.428781] LustreError: 370632:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10310.434059] LustreError: 370632:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [10317.473515] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10317.478336] LustreError: 371030:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10317.499257] LustreError: 371030:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10317.507666] LustreError: 371030:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10317.519390] LustreError: 371030:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10317.524164] LustreError: 371030:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [10338.198825] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [10342.493797] Key type lgssc unregistered [10342.782492] LNet: 372033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10342.788636] LNetError: 372033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10342.804247] LNet: Removed LNI 192.168.206.144@tcp [10343.934826] Key type .llcrypt unregistered [10343.936683] Key type ._llcrypt unregistered [10366.500822] Key type ._llcrypt registered [10366.504321] Key type .llcrypt registered [10366.660970] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [10382.894354] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [10383.997026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10384.166923] alg: No test for adler32 (adler32-zlib) [10385.187078] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [10385.494429] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [10387.159422] Key type lgssc registered [10388.296829] Lustre: Echo OBD driver; http://www.lustre.org/ [10424.144917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10424.182182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10425.415665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10425.439342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10425.524803] Lustre: lustre-MDT0000: new disk, initializing [10425.607384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10425.625565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10429.657428] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10438.814807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10438.932729] Lustre: 375192:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10438.998484] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10439.002822] Lustre: Skipped 1 previous similar message [10439.082906] Lustre: lustre-MDT0001: new disk, initializing [10439.174628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10439.211996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10439.231268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10442.997260] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10451.573884] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10456.395506] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10462.005973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10462.223598] Lustre: lustre-OST0000: new disk, initializing [10462.226570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10462.235423] Lustre: 377161:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10462.325913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10467.984468] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10471.996185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10472.007812] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10472.090163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10476.502691] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10480.840949] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10481.027302] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10485.920311] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10486.121962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10492.897737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10492.900264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10492.905485] LustreError: Skipped 1 previous similar message [10492.919427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10492.925226] Lustre: Skipped 1 previous similar message [10494.409263] Lustre: server umount lustre-OST0000 complete [10498.020249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10498.043981] Lustre: Skipped 1 previous similar message [10498.051406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10498.064604] Lustre: Skipped 1 previous similar message [10503.142502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10503.145417] Lustre: Skipped 1 previous similar message [10508.260352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10508.272305] Lustre: Skipped 1 previous similar message [10512.351113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10512.366829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10512.410412] Lustre: Skipped 1 previous similar message [10512.830048] Lustre: server umount lustre-MDT0000 complete [10517.508801] LustreError: 375187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789093168 with bad export cookie 315878738190833696 [10517.513273] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10517.524434] LustreError: 375187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10517.962631] Lustre: server umount lustre-MDT0001 complete [10530.338472] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10532.272771] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 22:19:41 (1789093181) [10539.459679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10539.927669] LustreError: 379620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10539.974913] LustreError: 379620:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10540.025605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10545.129448] LustreError: 379621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10545.148891] LustreError: 379621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10545.226921] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10549.227381] LustreError: 379620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10553.477575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10553.797666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10559.129609] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10566.886462] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10572.960754] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10580.408099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10580.830569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10587.051037] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10595.784031] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10604.227421] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10604.445426] Lustre: lustre-OST0001: new disk, initializing [10604.451927] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10604.467660] Lustre: 382460:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10604.607208] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10610.662977] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10612.861233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10612.874995] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10613.014795] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10620.494987] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10628.395660] Lustre: lustre-OST0001: Not available for connect from 192.168.206.44@tcp (stopping) [10628.577552] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10628.584471] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10628.589930] LustreError: Skipped 1 previous similar message [10628.597084] Lustre: Skipped 1 previous similar message [10631.329336] Lustre: server umount lustre-OST0001 complete [10633.486460] LustreError: 381445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10638.823651] LustreError: 382188:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10638.856876] LustreError: 382188:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10649.059109] LustreError: 381444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10649.077181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10649.084801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10649.084808] Lustre: Skipped 2 previous similar messages [10649.127279] LustreError: 381444:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10649.132740] Lustre: Skipped 2 previous similar messages [10652.883660] Lustre: server umount lustre-OST0000 complete [10659.321930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10659.372217] Lustre: Skipped 1 previous similar message [10663.874638] LustreError: 383693:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (000000006a542595) refcount nonzero (1) after lock cleanup; forcing cleanup. [10663.923975] Lustre: server umount lustre-MDT0000 complete [10668.015505] LustreError: 379602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789093318 with bad export cookie 315878738190834578 [10668.019119] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10668.037606] LustreError: 379602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10674.468485] Lustre: server umount lustre-MDT0001 complete [10685.312600] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [10688.873537] Key type lgssc unregistered [10689.229075] LNet: 384447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10689.244046] LNetError: 384447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10689.260718] LNet: Removed LNI 192.168.206.144@tcp [10690.376416] Key type .llcrypt unregistered [10690.379351] Key type ._llcrypt unregistered [10747.572707] Key type ._llcrypt registered [10747.574011] Key type .llcrypt registered [10747.714517] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [10748.640938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10748.695665] alg: No test for adler32 (adler32-zlib) [10749.877480] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [10750.245305] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [10751.896074] Key type lgssc registered [10753.201113] Lustre: Echo OBD driver; http://www.lustre.org/ [10761.134584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10761.152209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10762.537959] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10762.566084] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10762.802193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10768.348674] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10777.191500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10777.356290] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10777.408157] Lustre: 387251:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10777.664477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10782.358658] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10789.644317] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10794.826271] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10798.048395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10798.055379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10798.068292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10799.072600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10799.073977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10802.806669] Lustre: server umount lustre-MDT0000 complete [10805.747809] LustreError: 387244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789093456 with bad export cookie 10592826524884790699 [10805.749771] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10805.758629] LustreError: 387244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10805.982680] Lustre: server umount lustre-MDT0001 complete [10812.733066] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 22:24:22 (1789093462) [10817.977920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10818.504043] LustreError: 389766:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10818.513834] LustreError: 389766:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10818.550686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10821.893753] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10823.648268] LustreError: 389767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10828.399332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10828.732837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10832.158367] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10839.154473] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10844.071465] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10849.651349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10849.825907] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10850.052249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10855.125671] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10860.709611] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10868.086135] Lustre: Permanently deactivating lustre-OST0000 [10868.109101] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10876.948377] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10879.615784] LustreError: 392799:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10879.624952] LustreError: 392799:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [10879.630483] LustreError: 392799:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10881.721165] LustreError: 392847:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [10881.731884] LustreError: 392847:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10889.894658] Lustre: server umount lustre-OST0000 complete [10892.642576] LustreError: 393096:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [10892.650683] LustreError: 393096:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10895.328769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10895.346188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10895.383586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10895.395020] Lustre: Skipped 1 previous similar message [10898.921312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10898.928295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10904.034675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10904.046953] Lustre: Skipped 2 previous similar messages [10905.170786] Lustre: server umount lustre-MDT0000 complete [10909.155583] LustreError: 389767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10909.168109] LustreError: 389767:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10909.240205] LustreError: 391597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789093560 with bad export cookie 10592826524884791280 [10909.240896] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10909.262401] LustreError: 391597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10920.055306] Lustre: server umount lustre-MDT0001 complete [10926.945662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10930.718788] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10932.998359] LustreError: 394256:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10938.131947] Lustre: 394595:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.144@tcp [10939.881064] Lustre: server umount MGS complete [10949.906631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10950.194139] LustreError: 395319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10950.209971] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10950.213643] Lustre: Skipped 1 previous similar message [10950.253290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10953.644958] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10955.748883] LustreError: 395320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10963.005053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10963.516949] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10968.407188] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10975.906922] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10981.122394] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10982.836844] Lustre: Permanently reactivating lustre-OST0000 [10982.855481] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10982.861327] Lustre: Skipped 2 previous similar messages [11010.527151] LustreError: 397099:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11017.126410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11017.341802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11017.346882] Lustre: Skipped 1 previous similar message [11022.929125] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11028.484339] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11036.131888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11036.142605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11036.144089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11036.159744] Lustre: Skipped 1 previous similar message [11039.145516] Lustre: server umount lustre-OST0000 complete [11045.345268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11045.351115] LustreError: Skipped 1 previous similar message [11045.356681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11045.363147] Lustre: Skipped 1 previous similar message [11047.852655] Lustre: server umount lustre-MDT0000 complete [11050.363337] LustreError: 395301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789093701 with bad export cookie 10592826524884792330 [11050.371414] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11050.379397] LustreError: 395301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11050.590801] Lustre: server umount lustre-MDT0001 complete [11058.930579] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [11060.922676] Key type lgssc unregistered [11061.129873] LNet: 399730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11061.135812] LNetError: 399730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11061.149220] LNet: Removed LNI 192.168.206.144@tcp [11061.700089] Key type .llcrypt unregistered [11061.704901] Key type ._llcrypt unregistered [11077.021764] Key type ._llcrypt registered [11077.024249] Key type .llcrypt registered [11077.118512] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [11089.277613] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [11089.861648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11089.884946] alg: No test for adler32 (adler32-zlib) [11090.965661] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [11091.147187] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [11092.791202] Key type lgssc registered [11093.564730] Lustre: Echo OBD driver; http://www.lustre.org/ [11129.372504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11129.403928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11130.706807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11130.782777] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11130.914531] Lustre: lustre-MDT0000: new disk, initializing [11131.065300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11131.111924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11136.509971] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11147.396394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11147.644350] Lustre: 402867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11147.754147] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11147.760955] Lustre: Skipped 1 previous similar message [11147.852326] Lustre: lustre-MDT0001: new disk, initializing [11147.939892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11147.991966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11148.023684] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11152.169947] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11161.672778] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11167.596961] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11175.163644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11175.375960] Lustre: lustre-OST0000: new disk, initializing [11175.380750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11175.389312] Lustre: 404836:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11175.513453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11182.903250] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11183.735877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11183.746184] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11183.852936] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11194.238742] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11199.192407] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11199.378909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11204.072451] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11204.250250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11209.710061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11209.724081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11209.725095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11209.729781] Lustre: Skipped 1 previous similar message [11209.754470] LustreError: Skipped 1 previous similar message [11212.676382] Lustre: server umount lustre-OST0000 complete [11219.940409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11219.943258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11219.947590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11219.970917] Lustre: Skipped 1 previous similar message [11219.986761] Lustre: Skipped 2 previous similar messages [11222.422931] Lustre: server umount lustre-MDT0000 complete [11226.399708] LustreError: 402858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789093877 with bad export cookie 2986182785104069470 [11226.401683] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11226.417329] LustreError: 402858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11226.769840] Lustre: server umount lustre-MDT0001 complete [11236.254397] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11238.509595] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 22:31:27 (1789093887) [11248.474654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11249.360161] LustreError: 407297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11249.392811] LustreError: 407297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11249.524594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11254.753139] LustreError: 407298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11256.820837] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11259.885619] LustreError: 407297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11265.007961] LustreError: 407298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11267.226198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11267.659166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11272.464277] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11279.518433] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11284.467118] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11290.752102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11291.317323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11297.643597] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11304.552758] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11306.786568] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11309.580550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11318.243546] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11318.257495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11318.274864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11320.294481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11320.313555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11320.328861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11321.890855] Lustre: server umount lustre-OST0000 complete [11328.994993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11329.013392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11329.042381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11329.049246] Lustre: Skipped 1 previous similar message [11331.838882] Lustre: server umount lustre-MDT0000 complete [11335.654353] LustreError: 408028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11335.685496] LustreError: 408028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11335.753568] LustreError: 407279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789093986 with bad export cookie 2986182785104070352 [11335.754552] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11335.761536] LustreError: 407279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11336.196328] Lustre: server umount lustre-MDT0001 complete [11346.768243] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [11349.271719] Key type lgssc unregistered [11349.555964] LNet: 411022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11349.567309] LNetError: 411022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11349.583292] LNet: Removed LNI 192.168.206.144@tcp [11350.361917] Key type .llcrypt unregistered [11350.365603] Key type ._llcrypt unregistered [11372.426640] Key type ._llcrypt registered [11372.429511] Key type .llcrypt registered [11372.640607] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [11372.884612] Key type .llcrypt unregistered [11372.887291] Key type ._llcrypt unregistered [11389.923641] Key type ._llcrypt registered [11389.925433] Key type .llcrypt registered [11390.091726] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [11390.923182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11390.993382] alg: No test for adler32 (adler32-zlib) [11391.996580] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [11392.209270] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [11393.831178] Key type lgssc registered [11394.860677] Lustre: Echo OBD driver; http://www.lustre.org/ [11402.110153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11402.141232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11403.611262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11408.026730] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11413.989164] LustreError: 413236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11414.023951] LustreError: 413236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11416.309496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11416.782359] LustreError: 413237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11416.804447] LustreError: 413237:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11416.826436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11422.177349] LustreError: 413236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11422.202473] LustreError: 413236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11422.461496] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11427.310056] LustreError: 413237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11427.331776] LustreError: 413237:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11428.341708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11429.516492] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11433.446334] LustreError: 413237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11433.477848] LustreError: 413237:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11436.769318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11437.196068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11441.180218] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11451.084951] Lustre: server umount lustre-OST0000 complete [11456.996886] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11457.013682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11457.032929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11458.018202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11458.020185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11458.041211] Lustre: Skipped 1 previous similar message [11463.137484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11463.143498] Lustre: Skipped 1 previous similar message [11464.268496] Lustre: server umount lustre-MDT0000 complete [11468.255364] Lustre: 412547:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789094103/real 1789094103] req@00000000404a9ed9 x1876001075258496/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789094119 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11468.263029] LustreError: 413231:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11468.272566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11468.284470] LustreError: 413231:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11468.351209] LustreError: 413216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789094119 with bad export cookie 12020967742515945907 [11468.365473] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11468.366423] LustreError: 413216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11468.707467] Lustre: server umount lustre-MDT0001 complete [11481.433025] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [11484.873297] Key type lgssc unregistered [11485.120044] LNet: 416381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11485.133280] LNetError: 416381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11485.158689] LNet: Removed LNI 192.168.206.144@tcp [11485.965463] Key type .llcrypt unregistered [11485.967688] Key type ._llcrypt unregistered [11512.165628] Key type ._llcrypt registered [11512.179315] Key type .llcrypt registered [11512.402150] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [11514.605369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11514.742160] alg: No test for adler32 (adler32-zlib) [11516.094959] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [11516.559967] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [11518.216130] Key type lgssc registered [11519.830200] Lustre: Echo OBD driver; http://www.lustre.org/ [11527.993371] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11528.005934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11543.520225] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11548.640169] LustreError: 417773:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.144@tcp: failed processing log, type 4: rc = -110 [11575.264186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11581.227180] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11588.480328] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11595.730188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11596.776318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11596.876062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11601.864878] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11601.892511] LustreError: 417797:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11601.923743] LustreError: 417797:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11607.019573] LustreError: 417798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11610.091312] LustreError: 417797:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11610.120731] LustreError: 417797:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11612.750613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11613.137798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11619.130056] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11628.093643] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11633.375527] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11638.761334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11638.762190] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11638.763805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11638.780824] Lustre: Skipped 1 previous similar message [11643.883560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11643.893947] Lustre: Skipped 1 previous similar message [11644.581132] Lustre: server umount lustre-OST0000 complete [11648.994621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11648.995797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11649.011934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11649.021749] Lustre: Skipped 2 previous similar messages [11654.112220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11654.117531] Lustre: Skipped 1 previous similar message [11654.532800] Lustre: server umount lustre-MDT0000 complete [11657.946661] LustreError: 417779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789094308 with bad export cookie 8844505327701157108 [11657.964872] LustreError: 417779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11657.969482] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11658.295551] Lustre: server umount lustre-MDT0001 complete [11668.483927] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [11671.940910] Key type lgssc unregistered [11672.300466] LNet: 421342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11672.310259] LNetError: 421342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11672.330499] LNet: Removed LNI 192.168.206.144@tcp [11673.291797] Key type .llcrypt unregistered [11673.293835] Key type ._llcrypt unregistered [11702.168248] Key type ._llcrypt registered [11702.171656] Key type .llcrypt registered [11702.332645] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [11703.580001] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11703.608658] alg: No test for adler32 (adler32-zlib) [11704.814357] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [11705.091745] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [11706.847246] Key type lgssc registered [11708.266642] Lustre: Echo OBD driver; http://www.lustre.org/ [11719.125466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11719.163279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11721.098960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11728.680395] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11730.401046] LustreError: 422763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11730.429889] LustreError: 422763:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11735.524558] LustreError: 422764:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11740.449623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11740.645459] LustreError: 422763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11741.313503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11747.090550] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11756.668680] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11762.747800] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11769.689954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11770.028277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11775.871536] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11781.794408] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11790.523908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11797.472777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11797.481526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11797.497289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11798.816455] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.44@tcp (stopping) [11801.058847] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11801.062029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11801.077988] Lustre: Skipped 1 previous similar message [11801.082271] Lustre: Skipped 1 previous similar message [11802.476975] Lustre: server umount lustre-MDT0000 complete [11803.927145] LustreError: 425253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11804.942299] LustreError: 422746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.44@tcp arrived at 1789094455 with bad export cookie 16893457791223285201 [11806.846545] LustreError: 422744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789094457 with bad export cookie 16893457791223284543 [11806.858574] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11806.861879] LustreError: 422744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11807.207102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11807.212376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11807.231987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11807.243217] Lustre: Skipped 1 previous similar message [11811.296179] LustreError: 425203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11811.301242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11811.316742] LustreError: 425203:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11813.069753] Lustre: server umount lustre-OST0000 complete [11819.230650] Lustre: server umount lustre-MDT0001 complete [11830.200948] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [11832.669581] Key type lgssc unregistered [11832.976099] LNet: 426471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11832.981510] LNetError: 426471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11833.009894] LNet: Removed LNI 192.168.206.144@tcp [11833.747745] Key type .llcrypt unregistered [11833.750600] Key type ._llcrypt unregistered [11858.373700] Key type ._llcrypt registered [11858.379995] Key type .llcrypt registered [11858.554981] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [11859.693661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11859.731582] alg: No test for adler32 (adler32-zlib) [11860.867671] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [11861.127814] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [11862.807178] Key type lgssc registered [11863.990938] Lustre: Echo OBD driver; http://www.lustre.org/ [11872.442456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11872.468422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11873.985923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11879.834612] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11883.493221] LustreError: 427890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11883.507785] LustreError: 427890:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11888.113952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11888.535986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11893.531808] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11901.160308] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11907.183896] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11914.640907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11915.100960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11918.203619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11923.230615] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11933.891847] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11938.795828] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11938.822492] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11940.329156] Lustre: server umount lustre-MDT0001 complete [11941.178636] LustreError: 427914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11943.907224] LustreError: 427890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11943.921906] LustreError: 427890:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11949.025514] LustreError: 429277:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11949.033417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11949.049398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11949.056976] Lustre: Skipped 1 previous similar message [11949.061804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11949.066411] Lustre: Skipped 1 previous similar message [11952.124755] Lustre: server umount lustre-OST0000 complete [11954.146256] LustreError: 427889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11958.296571] Lustre: server umount lustre-MDT0000 complete [11971.343317] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [11974.215931] Key type lgssc unregistered [11974.584847] LNet: 431573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11974.596449] LNetError: 431573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11974.624572] LNet: Removed LNI 192.168.206.144@tcp [11975.667133] Key type .llcrypt unregistered [11975.672950] Key type ._llcrypt unregistered [11999.033720] Key type ._llcrypt registered [11999.039836] Key type .llcrypt registered [11999.191206] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [12013.777835] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [12014.879196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12014.991033] alg: No test for adler32 (adler32-zlib) [12016.006888] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [12016.292164] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [12017.968625] Key type lgssc registered [12018.927252] Lustre: Echo OBD driver; http://www.lustre.org/ [12057.171630] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [12072.890503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12072.941913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12074.206313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12074.266490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12074.403122] Lustre: lustre-MDT0000: new disk, initializing [12074.496468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12074.513259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12078.674489] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [12091.363597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12091.580157] Lustre: lustre-OST0000: new disk, initializing [12091.583781] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12091.587601] Lustre: Skipped 1 previous similar message [12091.592769] Lustre: 436611:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12091.671948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12097.464588] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [12100.269580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12100.291195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12100.330099] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12105.927725] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12118.893874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12119.080264] Lustre: 435593:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12119.168711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12119.251162] Lustre: lustre-MDT0001: new disk, initializing [12119.352499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12119.397372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12119.406985] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12122.834125] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [12131.169343] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12131.368983] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12135.931685] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12136.128835] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12144.619948] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12144.627644] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12144.658712] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12144.694840] Lustre: Skipped 1 previous similar message [12144.720295] Lustre: Skipped 1 previous similar message [12145.477166] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.44@tcp (stopping) [12147.704105] Lustre: server umount lustre-MDT0001 complete [12154.852637] LustreError: 435601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12154.889583] LustreError: 435601:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12159.980561] LustreError: 436619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12160.015371] LustreError: 436619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12165.093951] LustreError: 438845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12165.120040] LustreError: 438845:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12170.211112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12170.218931] LustreError: 435604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12170.222564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12173.452382] Lustre: server umount lustre-OST0000 complete [12175.339918] LustreError: 438845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12183.529475] Lustre: server umount lustre-MDT0000 complete [12199.192293] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [12203.289446] Key type lgssc unregistered [12203.791755] LNet: 440355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12203.799780] LNetError: 440355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12203.821869] LNet: Removed LNI 192.168.206.144@tcp [12204.801096] Key type .llcrypt unregistered [12204.802826] Key type ._llcrypt unregistered [12243.565295] Key type ._llcrypt registered [12243.570398] Key type .llcrypt registered [12243.656375] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [12243.932334] Key type .llcrypt unregistered [12243.939278] Key type ._llcrypt unregistered [12259.181961] Key type ._llcrypt registered [12259.186234] Key type .llcrypt registered [12259.283629] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [12260.206795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12260.241690] alg: No test for adler32 (adler32-zlib) [12261.201176] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [12261.370418] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [12262.991231] Key type lgssc registered [12264.024897] Lustre: Echo OBD driver; http://www.lustre.org/ [12292.391745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12292.433666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12293.770621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12293.790594] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12293.929776] Lustre: lustre-MDT0000: new disk, initializing [12294.013315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12294.035441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12300.775487] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [12312.639486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12312.830232] Lustre: 443733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12312.921614] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12312.933728] Lustre: Skipped 1 previous similar message [12313.008266] Lustre: lustre-MDT0001: new disk, initializing [12313.124077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12313.153050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12313.162159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12317.886913] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [12327.765543] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12333.269207] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12338.979266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12339.201533] Lustre: lustre-OST0000: new disk, initializing [12339.205853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12339.213511] Lustre: 445704:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12339.339568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12345.065569] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [12346.463927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12346.475551] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12346.628216] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12353.990611] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12362.211047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12362.227051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12362.233044] Lustre: Skipped 1 previous similar message [12365.615393] Lustre: server umount lustre-MDT0000 complete [12368.946223] LustreError: 445703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789095019 with bad export cookie 3085867019545463551 [12368.950941] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12368.963290] LustreError: 445703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12369.366443] Lustre: server umount lustre-MDT0001 complete [12383.161617] Lustre: server umount lustre-OST0000 complete [12392.504868] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 22:50:42 (1789095042) [12400.287818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12400.746260] LustreError: 447730:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12400.767201] LustreError: 447730:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12400.856952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12405.503449] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [12406.244022] LustreError: 447731:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12410.339690] LustreError: 447731:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12414.391426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12414.844191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12419.405968] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [12427.200455] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12432.882561] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12452.649478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12452.817836] Lustre: Found index 0 for lustre-OST0000, updating log [12452.974994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12458.127400] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [12464.657907] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12471.412848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12482.021877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12482.022607] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12482.036300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12482.036309] Lustre: Skipped 2 previous similar messages [12482.041467] Lustre: Skipped 2 previous similar messages [12483.106356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12483.117035] Lustre: Skipped 2 previous similar messages [12486.244766] Lustre: server umount lustre-OST0000 complete [12501.984982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12501.992307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12502.015378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12504.555356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12509.666911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12509.680333] Lustre: Skipped 3 previous similar messages [12512.287336] LNet: 1 local NIs in recovery (showing 1): 192.168.206.144@tcp [12516.319118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12516.413739] Lustre: server umount lustre-MDT0000 complete [12516.837279] LustreError: 450244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12516.852917] LustreError: 450244:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12519.222424] LustreError: 447713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789095170 with bad export cookie 3085867019545464944 [12519.228268] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12519.233433] LustreError: 447713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12519.542372] Lustre: server umount lustre-MDT0001 complete [12527.394559] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 22:52:57 (1789095177) [13143.588306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13144.294605] LustreError: 454173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13144.427219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13148.845280] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [13149.671589] LustreError: 454174:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13154.784773] LustreError: 454173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13156.651505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13156.932461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13160.682032] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [13166.704256] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13171.781644] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13191.356168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13191.645589] Lustre: Found index 0 for lustre-OST0000, updating log [13191.947455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13194.040994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [13197.170831] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [13203.442528] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13203.488312] LNet: 1 local NIs in recovery (showing 1): 192.168.206.144@tcp [13206.116682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13206.127630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13206.133539] Lustre: Skipped 4 previous similar messages [13211.255857] Lustre: server umount lustre-OST0000 complete [13214.818533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13214.820038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13214.824537] Lustre: Skipped 1 previous similar message [13216.866673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13216.872944] Lustre: Skipped 4 previous similar messages [13219.978313] Lustre: server umount lustre-MDT0000 complete [13220.961033] LustreError: 455235:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13220.969432] LustreError: 455235:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13222.215436] LustreError: 454154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789095873 with bad export cookie 3085867019545466029 [13222.221288] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13222.222470] LustreError: 454154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13222.388431] Lustre: server umount lustre-MDT0001 complete [13249.466684] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [13257.091835] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [13268.000142] LNet: 1 local NIs in recovery (showing 1): 192.168.206.144@tcp [13287.305943] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 23:05:37 (1789095937) [13301.140822] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [13303.270974] Key type lgssc unregistered [13303.492214] LNet: 460785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13303.497251] LNetError: 460785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13407.982607] LNet: Removed LNI 192.168.206.144@tcp [13409.214639] Key type .llcrypt unregistered [13409.216469] Key type ._llcrypt unregistered [13427.292551] Key type ._llcrypt registered [13427.300953] Key type .llcrypt registered [13427.604749] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [13429.421172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13429.553335] alg: No test for adler32 (adler32-zlib) [13431.065275] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [13431.573045] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [13433.367559] Key type lgssc registered [13435.137694] Lustre: Echo OBD driver; http://www.lustre.org/ [13440.628400] LNet: Added LNI 192.168.206.144@tcp100 [8/256/0/180] [13442.003886] LNet: Added LNI 192.168.206.144@tcp101 [8/256/0/180] [13443.372671] LNet: Added LNI 192.168.206.144@tcp102 [8/256/0/180] [13444.920691] LNet: Added LNI 192.168.206.144@tcp103 [8/256/0/180] [13446.281895] LNet: Added LNI 192.168.206.144@tcp104 [8/256/0/180] [13447.226901] LNet: Added LNI 192.168.206.144@tcp105 [8/256/0/180] [13448.613046] LNet: Added LNI 192.168.206.144@tcp106 [8/256/0/180] [13449.741916] LNet: Added LNI 192.168.206.144@tcp107 [8/256/0/180] [13450.664322] LNet: Added LNI 192.168.206.144@tcp108 [8/256/0/180] [13451.510426] LNet: Added LNI 192.168.206.144@tcp109 [8/256/0/180] [13452.306903] LNet: Added LNI 192.168.206.144@tcp110 [8/256/0/180] [13453.325324] LNet: Added LNI 192.168.206.144@tcp111 [8/256/0/180] [13454.168362] LNet: Added LNI 192.168.206.144@tcp112 [8/256/0/180] [13454.951538] LNet: Added LNI 192.168.206.144@tcp113 [8/256/0/180] [13455.698458] LNet: Added LNI 192.168.206.144@tcp114 [8/256/0/180] [13456.505727] LNet: Added LNI 192.168.206.144@tcp115 [8/256/0/180] [13457.390277] LNet: Added LNI 192.168.206.144@tcp116 [8/256/0/180] [13458.371724] LNet: Added LNI 192.168.206.144@tcp117 [8/256/0/180] [13459.182150] LNet: Added LNI 192.168.206.144@tcp118 [8/256/0/180] [13459.909587] LNet: Added LNI 192.168.206.144@tcp119 [8/256/0/180] [13460.677979] LNet: Added LNI 192.168.206.144@tcp120 [8/256/0/180] [13461.680329] LNet: Added LNI 192.168.206.144@tcp121 [8/256/0/180] [13462.607692] LNet: Added LNI 192.168.206.144@tcp122 [8/256/0/180] [13463.370409] LNet: Added LNI 192.168.206.144@tcp123 [8/256/0/180] [13464.264225] LNet: Added LNI 192.168.206.144@tcp124 [8/256/0/180] [13465.043540] LNet: Added LNI 192.168.206.144@tcp125 [8/256/0/180] [13466.024969] LNet: Added LNI 192.168.206.144@tcp126 [8/256/0/180] [13466.957925] LNet: Added LNI 192.168.206.144@tcp127 [8/256/0/180] [13468.078075] LNet: Added LNI 192.168.206.144@tcp128 [8/256/0/180] [13469.019661] LNet: Added LNI 192.168.206.144@tcp129 [8/256/0/180] [13470.141193] LNet: Added LNI 192.168.206.144@tcp130 [8/256/0/180] [13471.012941] LNet: Added LNI 192.168.206.144@tcp131 [8/256/0/180] [13471.970458] LNet: Added LNI 192.168.206.144@tcp132 [8/256/0/180] [13473.488818] LNet: Added LNI 192.168.206.144@tcp133 [8/256/0/180] [13474.753853] LNet: Added LNI 192.168.206.144@tcp134 [8/256/0/180] [13476.273901] LNet: Added LNI 192.168.206.144@tcp135 [8/256/0/180] [13485.513647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13485.543376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13486.614662] Lustre: 463817:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.144@tcp: can't add NID 192.168.206.144@tcp130: rc = -75 [13486.667966] LNetError: 463841:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.144 [13486.676760] LNetError: Protocol error connecting to 192.168.206.144@tcp110 at host 192.168.206.144:988: is it running a compatible version of Lustre? [13486.783323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13486.844307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13486.879710] LNet: 7 local NIs in recovery (showing 7): 192.168.206.144@tcp121, 192.168.206.144@tcp123, 192.168.206.144@tcp124, 192.168.206.144@tcp125, 192.168.206.144@tcp122, 192.168.206.144@tcp128, 192.168.206.144@tcp127 [13486.951629] Lustre: lustre-MDT0000: new disk, initializing [13487.032923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13487.048224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13487.719697] LNetError: 463847:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.144 [13487.736176] LNetError: 463847:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [13487.749428] LNetError: Protocol error connecting to 192.168.206.144@tcp113 at host 192.168.206.144:988: is it running a compatible version of Lustre? [13487.761589] LNetError: Skipped 9 previous similar messages [13489.056204] LNetError: 463843:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.144 [13489.069192] LNetError: 463843:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [13489.087173] LNetError: Protocol error connecting to 192.168.206.144@tcp121 at host 192.168.206.144:988: is it running a compatible version of Lustre? [13489.109829] LNetError: Skipped 7 previous similar messages [13492.655221] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [13493.153665] LNetError: 463841:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.144 [13493.165056] LNetError: Protocol error connecting to 192.168.206.144@tcp121 at host 192.168.206.144:988: is it running a compatible version of Lustre? [13501.346891] LNetError: 461274:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.144 [13501.369675] LNetError: Protocol error connecting to 192.168.206.144@tcp121 at host 192.168.206.144:988: is it running a compatible version of Lustre? [13504.118824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13504.364835] Lustre: 463832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13504.431223] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13504.438135] Lustre: Skipped 1 previous similar message [13504.519986] Lustre: lustre-MDT0001: new disk, initializing [13504.579092] Lustre: 464636:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.144@tcp: can't add NID 192.168.206.144@tcp130: rc = -75 [13504.589721] Lustre: 464636:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13504.755366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13504.828091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13504.837833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13510.035480] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [13516.581269] LNetError: 463844:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.144 [13516.605906] LNetError: Protocol error connecting to 192.168.206.144@tcp110 at host 192.168.206.144:988: is it running a compatible version of Lustre? [13520.053885] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13526.469527] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13533.285112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13533.556679] Lustre: lustre-OST0000: new disk, initializing [13533.560228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13533.569060] Lustre: 465810:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13533.697748] Lustre: 465815:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.144@tcp: can't add NID 192.168.206.144@tcp130: rc = -75 [13533.703996] Lustre: 465815:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [13533.767716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13535.011051] LNetError: 463837:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.144 [13535.030039] LNetError: 463837:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [13535.054367] LNetError: Protocol error connecting to 192.168.206.144@tcp124 at host 192.168.206.144:988: is it running a compatible version of Lustre? [13535.067758] LNetError: Skipped 10 previous similar messages [13541.437026] Lustre: 466162:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.144@tcp: can't add NID 192.168.206.144@tcp130: rc = -75 [13541.442446] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [13541.457958] Lustre: 466162:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13541.597293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13541.607220] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13541.882762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13546.399375] LNet: 1 local NIs in recovery (showing 1): 192.168.206.144@tcp121 [13554.524228] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13567.456284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13567.457753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13567.469459] LustreError: Skipped 1 previous similar message [13567.480338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13567.480833] Lustre: Skipped 1 previous similar message [13567.495552] Lustre: Skipped 1 previous similar message [13567.905443] LNetError: 461276:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.144 [13567.941783] LNetError: 461276:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 12 previous similar messages [13567.963505] LNetError: Protocol error connecting to 192.168.206.144@tcp123 at host 192.168.206.144:988: is it running a compatible version of Lustre? [13567.979336] LNetError: Skipped 12 previous similar messages [13570.657444] Lustre: server umount lustre-OST0000 complete [13576.676153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13576.682331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13576.701045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13577.700086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13577.706354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13577.734438] Lustre: Skipped 1 previous similar message [13581.001024] Lustre: server umount lustre-MDT0000 complete [13585.196932] LustreError: 463825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789096236 with bad export cookie 16292719292841530 [13585.199857] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13585.207596] LustreError: 463825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13585.733462] Lustre: server umount lustre-MDT0001 complete [13599.132232] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [13603.559154] Key type lgssc unregistered [13603.945170] LNet: 467725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13603.950463] LNetError: 467725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13603.963613] LNet: Removed LNI 192.168.206.144@tcp [13605.023143] LNet: Removed LNI 192.168.206.144@tcp100 [13605.027837] LNetError: 467725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13605.038176] LNetError: 467725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13605.057049] LNet: Removed LNI 192.168.206.144@tcp101 [13606.111637] LNet: Removed LNI 192.168.206.144@tcp102 [13606.124177] LNetError: 467725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13606.136473] LNetError: 467725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13606.149986] LNet: Removed LNI 192.168.206.144@tcp103 [13607.199936] LNet: Removed LNI 192.168.206.144@tcp104 [13608.226648] LNet: Removed LNI 192.168.206.144@tcp105 [13608.238841] LNetError: 467725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13608.247215] LNetError: 467725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13608.263139] LNet: Removed LNI 192.168.206.144@tcp106 [13609.311183] LNet: Removed LNI 192.168.206.144@tcp107 [13610.337990] LNet: Removed LNI 192.168.206.144@tcp108 [13611.359240] LNet: Removed LNI 192.168.206.144@tcp109 [13612.384038] LNet: Removed LNI 192.168.206.144@tcp110 [13612.388200] LNetError: 467725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13612.403884] LNetError: 467725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13612.414208] LNet: Removed LNI 192.168.206.144@tcp111 [13613.471159] LNet: Removed LNI 192.168.206.144@tcp112 [13614.495086] LNet: Removed LNI 192.168.206.144@tcp113 [13615.519587] LNet: Removed LNI 192.168.206.144@tcp114 [13616.543152] LNet: Removed LNI 192.168.206.144@tcp115 [13617.567137] LNet: Removed LNI 192.168.206.144@tcp116 [13618.591153] LNet: Removed LNI 192.168.206.144@tcp117 [13619.615996] LNet: Removed LNI 192.168.206.144@tcp118 [13620.639239] LNet: Removed LNI 192.168.206.144@tcp119 [13620.641160] LNetError: 467725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13620.652978] LNetError: 467725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13620.663979] LNet: Removed LNI 192.168.206.144@tcp120 [13620.667927] LNet: Removed LNI 192.168.206.144@tcp121 [13621.727294] LNet: Removed LNI 192.168.206.144@tcp122 [13622.751408] LNet: Removed LNI 192.168.206.144@tcp123 [13623.775163] LNet: Removed LNI 192.168.206.144@tcp124 [13624.799258] LNet: Removed LNI 192.168.206.144@tcp125 [13625.826084] LNet: Removed LNI 192.168.206.144@tcp126 [13626.847127] LNet: Removed LNI 192.168.206.144@tcp127 [13627.873821] LNet: Removed LNI 192.168.206.144@tcp128 [13628.959285] LNet: Removed LNI 192.168.206.144@tcp129 [13628.961196] LNet: Removed LNI 192.168.206.144@tcp130 [13628.973267] LNet: Removed LNI 192.168.206.144@tcp131 [13628.980770] LNet: Removed LNI 192.168.206.144@tcp132 [13628.990173] LNet: Removed LNI 192.168.206.144@tcp133 [13629.005229] LNet: Removed LNI 192.168.206.144@tcp134 [13629.023777] LNet: Removed LNI 192.168.206.144@tcp135 [13630.018889] Key type .llcrypt unregistered [13630.024928] Key type ._llcrypt unregistered [13652.696432] Key type ._llcrypt registered [13652.698332] Key type .llcrypt registered [13652.864464] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [13667.553798] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [13668.434815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13668.504066] alg: No test for adler32 (adler32-zlib) [13669.593615] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [13669.860228] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [13671.559219] Key type lgssc registered [13672.920612] Lustre: Echo OBD driver; http://www.lustre.org/ [13709.032905] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 23:12:38 (1789096358) [13727.890891] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [13730.324390] Key type lgssc unregistered [13730.590216] LNet: 471816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13730.598717] LNetError: 471816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13730.611729] LNet: Removed LNI 192.168.206.144@tcp [13731.350725] Key type .llcrypt unregistered [13731.354616] Key type ._llcrypt unregistered [13746.822233] Key type ._llcrypt registered [13746.824617] Key type .llcrypt registered [13746.925534] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [13747.788723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13747.825236] alg: No test for adler32 (adler32-zlib) [13748.922476] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [13749.110865] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [13750.735246] Key type lgssc registered [13751.946797] Lustre: Echo OBD driver; http://www.lustre.org/ [13761.991668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13762.035108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13763.427142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13763.477913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13763.586815] Lustre: lustre-MDT0000: new disk, initializing [13763.660803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13763.679029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13768.174310] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [13779.124942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13779.285463] Lustre: 473043:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13779.384263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13779.394584] Lustre: Skipped 1 previous similar message [13779.474807] Lustre: lustre-MDT0001: new disk, initializing [13779.567461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13779.607046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13779.613864] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13784.012202] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [13793.725485] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13799.830311] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13807.257279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13807.526744] Lustre: lustre-OST0000: new disk, initializing [13807.530160] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13807.536286] Lustre: 475014:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13807.617175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13813.816161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13813.825825] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [13813.828181] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13813.969739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13822.985280] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13827.394947] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing lnet_if_list [13833.171472] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13833.192704] LNet: Added LNI 192.168.206.144@tcp10 [8/256/0/180] [13839.129919] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13839.172777] LNet: Added LNI 192.168.206.144@tcp11 [8/256/0/180] [13844.630604] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13844.670376] LNet: Added LNI 192.168.206.144@tcp12 [8/256/0/180] [13850.194223] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13850.221977] LNet: Added LNI 192.168.206.144@tcp13 [8/256/0/180] [13855.507201] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13855.536501] LNet: Added LNI 192.168.206.144@tcp14 [8/256/0/180] [13860.514219] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13860.541466] LNet: Added LNI 192.168.206.144@tcp15 [8/256/0/180] [13865.938931] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13865.967113] LNet: Added LNI 192.168.206.144@tcp16 [8/256/0/180] [13870.892594] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13870.918814] LNet: Added LNI 192.168.206.144@tcp17 [8/256/0/180] [13875.540386] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13875.572111] LNet: Added LNI 192.168.206.144@tcp18 [8/256/0/180] [13880.645459] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13880.682613] LNet: Added LNI 192.168.206.144@tcp19 [8/256/0/180] [13886.101626] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13886.121582] LNet: Added LNI 192.168.206.144@tcp20 [8/256/0/180] [13891.135799] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13891.161954] LNet: Added LNI 192.168.206.144@tcp21 [8/256/0/180] [13896.217396] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13896.256863] LNet: Added LNI 192.168.206.144@tcp22 [8/256/0/180] [13901.399922] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13901.417729] LNet: Added LNI 192.168.206.144@tcp23 [8/256/0/180] [13906.360435] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13906.379536] LNet: Added LNI 192.168.206.144@tcp24 [8/256/0/180] [13912.264860] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13912.295609] LNet: Added LNI 192.168.206.144@tcp25 [8/256/0/180] [13918.046498] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13918.087918] LNet: Added LNI 192.168.206.144@tcp26 [8/256/0/180] [13923.846405] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13923.889271] LNet: Added LNI 192.168.206.144@tcp27 [8/256/0/180] [13930.462905] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13930.501936] LNet: Added LNI 192.168.206.144@tcp28 [8/256/0/180] [13936.121481] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13936.148390] LNet: Added LNI 192.168.206.144@tcp29 [8/256/0/180] [13941.584471] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13941.619373] LNet: Added LNI 192.168.206.144@tcp30 [8/256/0/180] [13947.942816] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13947.974457] LNet: Added LNI 192.168.206.144@tcp31 [8/256/0/180] [13954.192733] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13954.216971] LNet: Added LNI 192.168.206.144@tcp32 [8/256/0/180] [13960.057775] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13960.084560] LNet: Added LNI 192.168.206.144@tcp33 [8/256/0/180] [13966.851684] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13966.893303] LNet: Added LNI 192.168.206.144@tcp34 [8/256/0/180] [13974.090494] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13974.118935] LNet: Added LNI 192.168.206.144@tcp35 [8/256/0/180] [13981.776408] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13981.808896] LNet: Added LNI 192.168.206.144@tcp36 [8/256/0/180] [13989.389878] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13989.417966] LNet: Added LNI 192.168.206.144@tcp37 [8/256/0/180] [13997.251451] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13997.275654] LNet: Added LNI 192.168.206.144@tcp38 [8/256/0/180] [14005.103801] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14005.149031] LNet: Added LNI 192.168.206.144@tcp39 [8/256/0/180] [14013.398964] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14013.433647] LNet: Added LNI 192.168.206.144@tcp40 [8/256/0/180] [14022.363232] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14022.401648] LNet: Added LNI 192.168.206.144@tcp41 [8/256/0/180] [14030.427203] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14030.468886] LNet: Added LNI 192.168.206.144@tcp42 [8/256/0/180] [14037.863047] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14037.889571] LNet: Added LNI 192.168.206.144@tcp43 [8/256/0/180] [14046.165316] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14046.213715] LNet: Added LNI 192.168.206.144@tcp44 [8/256/0/180] [14054.692977] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14054.724105] LNet: Added LNI 192.168.206.144@tcp45 [8/256/0/180] [14063.454278] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14063.509766] LNet: Added LNI 192.168.206.144@tcp46 [8/256/0/180] [14071.734622] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14071.761404] LNet: Added LNI 192.168.206.144@tcp47 [8/256/0/180] [14081.717569] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14081.758413] LNet: Added LNI 192.168.206.144@tcp48 [8/256/0/180] [14090.020137] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14090.048574] LNet: Added LNI 192.168.206.144@tcp49 [8/256/0/180] [14098.051436] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14098.075927] LNet: Added LNI 192.168.206.144@tcp50 [8/256/0/180] [14116.321588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14116.324994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14116.334502] Lustre: Skipped 1 previous similar message [14116.343656] Lustre: Skipped 1 previous similar message [14121.324467] Lustre: server umount lustre-MDT0000 complete [14121.443881] LustreError: 473055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14121.459718] LustreError: 473055:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14125.021285] LustreError: 473038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789096775 with bad export cookie 7866823310922641197 [14125.024871] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14125.030665] LustreError: 473038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14125.419800] Lustre: server umount lustre-MDT0001 complete [14142.948768] Lustre: 472317:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789096777/real 1789096777] req@000000009328afda x1876003546898432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789096793 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14142.974450] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14142.991831] Lustre: Skipped 1 previous similar message [14143.041153] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [14147.039598] Lustre: 472316:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789096781/real 1789096781] req@00000000f0b67cf1 x1876003546898560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789096797 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14148.063500] Lustre: 472318:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789096782/real 1789096782] req@0000000058e1bcea x1876003546898688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789096798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14150.177983] Lustre: 472318:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789096785/real 1789096785] req@00000000b41ecb36 x1876003546898944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789096801 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14151.142807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14155.231394] Lustre: 472316:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789096790/real 1789096790] req@00000000bf9748a8 x1876003546899072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789096806 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14169.840484] LustreError: 475007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14169.895031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14172.901466] LustreError: 475008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14173.926948] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14174.055712] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14179.298566] LustreError: 475007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14179.315180] LustreError: 475007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14181.828132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14182.258250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14187.400965] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14187.575661] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14195.538377] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14202.716922] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14223.330780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14223.336529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14223.337117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14223.341689] Lustre: Skipped 1 previous similar message [14223.364302] Lustre: Skipped 2 previous similar messages [14228.131913] Lustre: server umount lustre-OST0000 complete [14233.568832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14233.572873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14233.587963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14233.613199] Lustre: Skipped 1 previous similar message [14238.689563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14238.697288] Lustre: Skipped 2 previous similar messages [14243.811785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14243.820374] Lustre: Skipped 1 previous similar message [14246.367235] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14246.601956] Lustre: server umount lustre-MDT0000 complete [14248.932517] LustreError: 499258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14248.966105] LustreError: 499258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14250.535898] LustreError: 473036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789096901 with bad export cookie 7866823310922643794 [14250.539576] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14250.552821] LustreError: 473036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14251.138898] Lustre: server umount lustre-MDT0001 complete [14262.817240] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [14265.831366] Key type lgssc unregistered [14266.236068] LNet: 502042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14266.246559] LNetError: 502042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14267.295651] LNet: Removed LNI 192.168.206.144@tcp [14267.303837] LNetError: 502042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14267.318037] LNet: Removed LNI 192.168.206.144@tcp10 [14267.322661] LNet: Removed LNI 192.168.206.144@tcp11 [14267.332566] LNet: Removed LNI 192.168.206.144@tcp12 [14267.339652] LNet: Removed LNI 192.168.206.144@tcp13 [14267.346641] LNet: Removed LNI 192.168.206.144@tcp14 [14267.356834] LNet: Removed LNI 192.168.206.144@tcp15 [14267.370764] LNet: Removed LNI 192.168.206.144@tcp16 [14267.378581] LNet: Removed LNI 192.168.206.144@tcp17 [14267.380655] LNet: Removed LNI 192.168.206.144@tcp18 [14267.385509] LNet: Removed LNI 192.168.206.144@tcp19 [14267.389446] LNet: Removed LNI 192.168.206.144@tcp20 [14267.391994] LNet: Removed LNI 192.168.206.144@tcp21 [14267.399671] LNet: Removed LNI 192.168.206.144@tcp22 [14267.405883] LNet: Removed LNI 192.168.206.144@tcp23 [14267.410063] LNet: Removed LNI 192.168.206.144@tcp24 [14267.411927] LNet: Removed LNI 192.168.206.144@tcp25 [14267.422985] LNet: Removed LNI 192.168.206.144@tcp26 [14267.427524] LNet: Removed LNI 192.168.206.144@tcp27 [14267.432988] LNet: Removed LNI 192.168.206.144@tcp28 [14267.436477] LNet: Removed LNI 192.168.206.144@tcp29 [14267.444424] LNet: Removed LNI 192.168.206.144@tcp30 [14267.446795] LNet: Removed LNI 192.168.206.144@tcp31 [14267.460008] LNet: Removed LNI 192.168.206.144@tcp32 [14267.463910] LNet: Removed LNI 192.168.206.144@tcp33 [14267.476431] LNet: Removed LNI 192.168.206.144@tcp34 [14267.482560] LNet: Removed LNI 192.168.206.144@tcp35 [14267.486463] LNet: Removed LNI 192.168.206.144@tcp36 [14267.492356] LNet: Removed LNI 192.168.206.144@tcp37 [14267.495816] LNet: Removed LNI 192.168.206.144@tcp38 [14267.498770] LNet: Removed LNI 192.168.206.144@tcp39 [14267.501521] LNet: Removed LNI 192.168.206.144@tcp40 [14267.504330] LNet: Removed LNI 192.168.206.144@tcp41 [14267.506852] LNet: Removed LNI 192.168.206.144@tcp42 [14267.510273] LNet: Removed LNI 192.168.206.144@tcp43 [14267.513883] LNet: Removed LNI 192.168.206.144@tcp44 [14267.516740] LNet: Removed LNI 192.168.206.144@tcp45 [14267.519527] LNet: Removed LNI 192.168.206.144@tcp46 [14267.522303] LNet: Removed LNI 192.168.206.144@tcp47 [14267.524848] LNet: Removed LNI 192.168.206.144@tcp48 [14267.527428] LNet: Removed LNI 192.168.206.144@tcp49 [14267.542895] LNet: Removed LNI 192.168.206.144@tcp50 [14268.457340] Key type .llcrypt unregistered [14268.467198] Key type ._llcrypt unregistered [14290.089968] Key type ._llcrypt registered [14290.092257] Key type .llcrypt registered [14290.265829] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [14306.321693] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [14307.397394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14307.434955] alg: No test for adler32 (adler32-zlib) [14308.517292] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [14308.744986] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [14310.479191] Key type lgssc registered [14311.759072] Lustre: Echo OBD driver; http://www.lustre.org/ [14348.438873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14348.477176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14349.735149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14349.768843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14349.875556] Lustre: lustre-MDT0000: new disk, initializing [14350.054172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14350.082865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14355.004314] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14365.111834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14365.213467] Lustre: 505240:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14365.269314] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14365.272416] Lustre: Skipped 1 previous similar message [14365.307277] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [14365.352213] Lustre: lustre-MDT0001: new disk, initializing [14365.433303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14365.468228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14365.481560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14369.069791] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14376.853072] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14381.928268] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14388.264228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14388.428411] Lustre: *** cfs_fail_loc=90f, val=0*** [14388.501477] Lustre: lustre-OST0000: new disk, initializing [14388.504504] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14388.510604] Lustre: 507256:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14388.576433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14394.230241] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14398.013104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14398.022719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14398.132432] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14402.921909] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14413.794886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14413.795105] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14413.803063] LustreError: Skipped 1 previous similar message [14413.804173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14413.818968] Lustre: Skipped 1 previous similar message [14413.833565] Lustre: Skipped 1 previous similar message [14416.155869] Lustre: server umount lustre-OST0000 complete [14421.986199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14422.000790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14422.031806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14424.045996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14424.049940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14425.984195] Lustre: server umount lustre-MDT0000 complete [14429.396143] LustreError: 505233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789097080 with bad export cookie 7991607355923982062 [14429.405180] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14429.410688] LustreError: 505233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14429.781610] Lustre: server umount lustre-MDT0001 complete [14439.916718] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [14442.421721] Key type lgssc unregistered [14442.823662] LNet: 509162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14442.829616] LNetError: 509162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14442.845569] LNet: Removed LNI 192.168.206.144@tcp [14443.726106] Key type .llcrypt unregistered [14443.728228] Key type ._llcrypt unregistered [14461.124638] Key type ._llcrypt registered [14461.127175] Key type .llcrypt registered [14461.220838] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [14462.084650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14462.109977] alg: No test for adler32 (adler32-zlib) [14463.099720] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [14463.313194] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [14464.991181] Key type lgssc registered [14465.930190] Lustre: Echo OBD driver; http://www.lustre.org/ [14473.082930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14473.114446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14474.515185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14478.400285] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14484.962900] LustreError: 510426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14484.984427] LustreError: 510426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14486.944196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14487.268568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14491.458722] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14498.447994] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14503.824351] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14510.476683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14510.875200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14516.596356] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14523.433429] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14532.384107] loop0: detected capacity change from 0 to 204800000 [14532.537985] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14532.550842] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14532.565289] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14538.115759] loop0: detected capacity change from 0 to 204800000 [14538.278825] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14538.290848] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14538.311498] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14544.184415] loop0: detected capacity change from 0 to 204800000 [14549.560049] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14549.810543] Lustre: fs15246-OST0000: new disk, initializing [14549.816243] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14549.825489] Lustre: 514018:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14549.909975] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14556.116927] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14565.609222] loop1: detected capacity change from 0 to 204800000 [14571.616187] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14571.807330] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14571.941736] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14572.070208] Lustre: fs15246-MDT0000: new disk, initializing [14572.237498] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14572.276585] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14572.394280] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14573.483258] Lustre: 515141:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000b6a39da5 x1876004295456000/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789097240 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14577.746279] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14582.569316] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14582.590845] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14582.624278] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14601.711889] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14601.725206] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14606.066861] Lustre: server umount fs15246-MDT0000 complete [14606.821630] LustreError: 511666:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14611.942296] LustreError: 510420:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14614.844190] Lustre: server umount fs15246-OST0000 complete [14625.762947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14625.771155] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14625.787620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14627.307915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14627.311505] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14631.674515] Lustre: server umount lustre-OST0000 complete [14636.004410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14636.025866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14636.049768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14636.068441] Lustre: Skipped 1 previous similar message [14641.622887] Lustre: server umount lustre-MDT0000 complete [14642.656898] LustreError: 510426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14642.681595] LustreError: 510426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14646.472840] LustreError: 511160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789097297 with bad export cookie 16777427136562798933 [14646.485956] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14646.493625] LustreError: 511160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14647.121966] Lustre: server umount lustre-MDT0001 complete [14662.808697] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [14665.599658] Key type lgssc unregistered [14665.893531] LNet: 518238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14665.900586] LNetError: 518238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14665.912895] LNet: Removed LNI 192.168.206.144@tcp [14666.777188] Key type .llcrypt unregistered [14666.780979] Key type ._llcrypt unregistered [14687.517439] Key type ._llcrypt registered [14687.519774] Key type .llcrypt registered [14687.619736] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [14688.501983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14688.518631] alg: No test for adler32 (adler32-zlib) [14689.622448] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [14689.873330] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [14691.543181] Key type lgssc registered [14692.641563] Lustre: Echo OBD driver; http://www.lustre.org/ [14733.786067] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [14744.690693] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [14776.208314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14776.228059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14777.487650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14777.522905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14777.594204] Lustre: lustre-MDT0000: new disk, initializing [14777.657763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14777.674835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14781.065682] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14789.552673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14789.693173] Lustre: 522704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14789.774249] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14789.777679] Lustre: Skipped 1 previous similar message [14789.866578] Lustre: lustre-MDT0001: new disk, initializing [14789.986233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14790.044453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14790.049376] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14794.434464] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14802.165831] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14806.742670] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14812.334509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14812.614899] Lustre: lustre-OST0000: new disk, initializing [14812.622106] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14812.633267] Lustre: 524673:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14812.689699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14818.312434] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14818.378435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14818.389243] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14818.494581] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14826.892032] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14832.082413] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14832.373943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14836.618787] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14836.848729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14839.266296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14839.273071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14839.274996] LustreError: Skipped 1 previous similar message [14839.277797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14839.285733] Lustre: Skipped 1 previous similar message [14839.316394] Lustre: Skipped 1 previous similar message [14844.386075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14844.405299] Lustre: Skipped 1 previous similar message [14844.786798] Lustre: server umount lustre-OST0000 complete [14849.506058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14849.510573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14849.517907] Lustre: Skipped 1 previous similar message [14849.532056] Lustre: Skipped 1 previous similar message [14854.618101] Lustre: server umount lustre-MDT0000 complete [14858.245075] LustreError: 522698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789097509 with bad export cookie 10839870616445215877 [14858.246393] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14858.260212] LustreError: 522698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14858.653674] Lustre: server umount lustre-MDT0001 complete [14881.207604] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [14891.220465] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [14921.532136] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14669 sec ========= 23:32:51 (1789097571) [14923.154578] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:32:52 (1789097572) === [14926.666749] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:32:56 (1789097576) === [14964.062869] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [14966.472147] Key type lgssc unregistered [14966.798316] LNet: 531314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14966.804795] LNetError: 531314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14966.819178] LNet: Removed LNI 192.168.206.144@tcp [14967.630534] Key type .llcrypt unregistered [14967.633714] Key type ._llcrypt unregistered