[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 518455483 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002369] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007475] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009007] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010023] pid_max: default: 32768 minimum: 301 [ 0.011145] LSM: Security Framework initializing [ 0.012091] Yama: becoming mindful. [ 0.013047] SELinux: Initializing. [ 0.015031] *** VALIDATE selinux *** [ 0.023681] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028320] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030123] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031097] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032087] *** VALIDATE tmpfs *** [ 0.034098] *** VALIDATE proc *** [ 0.035156] *** VALIDATE cgroup *** [ 0.036007] *** VALIDATE cgroup2 *** [ 0.038244] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041028] Spectre V2 : User space: Vulnerable [ 0.042008] Speculative Store Bypass: Vulnerable [ 0.045410] debug: unmapping init [mem 0xffffffff8a659000-0xffffffff8a660fff] [ 0.048236] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049827] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050090] ... version: 2 [ 0.051020] ... bit width: 48 [ 0.052015] ... generic registers: 4 [ 0.053011] ... value mask: 0000ffffffffffff [ 0.054018] ... max period: 00007fffffffffff [ 0.055012] ... fixed-purpose events: 3 [ 0.056010] ... event mask: 000000070000000f [ 0.057324] rcu: Hierarchical SRCU implementation. [ 0.059679] smp: Bringing up secondary CPUs ... [ 0.060758] x86: Booting SMP configuration: [ 0.061023] .... node #0, CPUs: #1 #2 #3 [ 0.066010] smp: Brought up 1 node, 4 CPUs [ 0.068015] smpboot: Max logical packages: 1 [ 0.069011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.133064] node 0 deferred pages initialised in 62ms [ 0.141133] devtmpfs: initialized [ 0.142275] x86/mm: Memory block size: 128MB [ 0.146280] gcov: version magic: 0x41383552 [ 0.149168] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152191] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155442] pinctrl core: initialized pinctrl subsystem [ 0.157180] [ 0.157847] ************************************************************* [ 0.160019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162010] ** ** [ 0.165011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168011] ** ** [ 0.170010] ** This means that this kernel is built to expose internal ** [ 0.172021] ** IOMMU data structures, which may compromise security on ** [ 0.175053] ** your system. ** [ 0.178011] ** ** [ 0.180009] ** If you see this message and you are not debugging the ** [ 0.182012] ** kernel, report this immediately to your vendor! ** [ 0.184010] ** ** [ 0.186010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.188009] ************************************************************* [ 0.191613] NET: Registered protocol family 16 [ 0.193592] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.196053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.199069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.204007] cpuidle: using governor menu [ 0.205924] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.209547] PCI: Using configuration type 1 for base access [ 0.212141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.225489] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.227025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.231102] cryptd: max_cpu_qlen set to 1000 [ 0.234326] ACPI: Added _OSI(Module Device) [ 0.236020] ACPI: Added _OSI(Processor Device) [ 0.237025] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.240018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.247184] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.256641] ACPI: Interpreter enabled [ 0.259191] ACPI: PM: (supports S0 S3 S4 S5) [ 0.260010] ACPI: Using IOAPIC for interrupt routing [ 0.263117] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.268666] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.282506] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.285031] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.288013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.292100] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.299717] acpiphp: Slot [2] registered [ 0.301140] acpiphp: Slot [5] registered [ 0.303199] acpiphp: Slot [6] registered [ 0.306114] acpiphp: Slot [7] registered [ 0.308104] acpiphp: Slot [8] registered [ 0.310110] acpiphp: Slot [9] registered [ 0.311210] acpiphp: Slot [10] registered [ 0.313171] acpiphp: Slot [3] registered [ 0.315131] acpiphp: Slot [4] registered [ 0.317085] acpiphp: Slot [11] registered [ 0.319088] acpiphp: Slot [12] registered [ 0.320148] acpiphp: Slot [13] registered [ 0.323124] acpiphp: Slot [14] registered [ 0.324082] acpiphp: Slot [15] registered [ 0.326118] acpiphp: Slot [16] registered [ 0.327203] acpiphp: Slot [17] registered [ 0.329089] acpiphp: Slot [18] registered [ 0.331110] acpiphp: Slot [19] registered [ 0.332084] acpiphp: Slot [20] registered [ 0.334099] acpiphp: Slot [21] registered [ 0.336118] acpiphp: Slot [22] registered [ 0.337082] acpiphp: Slot [23] registered [ 0.339079] acpiphp: Slot [24] registered [ 0.341145] acpiphp: Slot [25] registered [ 0.343083] acpiphp: Slot [26] registered [ 0.344244] acpiphp: Slot [27] registered [ 0.346099] acpiphp: Slot [28] registered [ 0.348095] acpiphp: Slot [29] registered [ 0.350097] acpiphp: Slot [30] registered [ 0.352094] acpiphp: Slot [31] registered [ 0.353238] PCI host bridge to bus 0000:00 [ 0.355202] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.358055] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.361018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.365023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.368020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.371053] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.373250] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.377414] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.381432] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.391890] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.399014] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.402022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.403017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.407016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.410940] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.413945] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.416050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.419640] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.425013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.442014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.448013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.455601] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.467013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.481014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.503014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.522281] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.535014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.549022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.578013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.596000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.618098] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.639015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.669013] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.700000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.700000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.704000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.743013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.774649] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.796015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.813015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.854014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.877367] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.898022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.920018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.977804] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.013159] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.016398] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.019468] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.025741] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.074205] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.080254] iommu: Default domain type: Passthrough [ 1.086397] SCSI subsystem initialized [ 1.089145] ACPI: bus type USB registered [ 1.092087] usbcore: registered new interface driver usbfs [ 1.094086] usbcore: registered new interface driver hub [ 1.096064] usbcore: registered new device driver usb [ 1.099231] pps_core: LinuxPPS API ver. 1 registered [ 1.111012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.134044] PTP clock support registered [ 1.136272] EDAC MC: Ver: 3.0.0 [ 1.143164] PCI: Using ACPI for IRQ routing [ 1.144490] NetLabel: Initializing [ 1.150015] NetLabel: domain hash size = 128 [ 1.152009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.157119] NetLabel: unlabeled traffic allowed by default [ 1.161421] vgaarb: loaded [ 1.163384] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.165011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.173569] clocksource: Switched to clocksource kvm-clock [ 1.293589] VFS: Disk quotas dquot_6.6.0 [ 1.295347] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.298113] *** VALIDATE ramfs *** [ 1.299502] *** VALIDATE hugetlbfs *** [ 1.301194] pnp: PnP ACPI init [ 1.304175] pnp: PnP ACPI: found 6 devices [ 1.329971] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.334279] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.337900] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.345303] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.347760] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.350089] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.352840] NET: Registered protocol family 2 [ 1.355397] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.360529] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.364279] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.369915] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.373344] TCP: Hash tables configured (established 65536 bind 65536) [ 1.376357] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.379572] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.383934] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.387140] NET: Registered protocol family 1 [ 1.389589] RPC: Registered named UNIX socket transport module. [ 1.391917] RPC: Registered udp transport module. [ 1.393556] RPC: Registered tcp transport module. [ 1.395131] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.397388] NET: Registered protocol family 44 [ 1.399098] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.401599] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.403871] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.406220] PCI: CLS 0 bytes, default 64 [ 1.408713] Unpacking initramfs... [ 3.094960] debug: unmapping init [mem 0xffff9af6bcc54000-0xffff9af6bffbffff] [ 3.099111] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.101228] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.104115] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.442559] Initialise system trusted keyrings [ 4.445491] Key type blacklist registered [ 4.456494] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.475309] zbud: loaded [ 4.485898] *** VALIDATE nfs *** [ 4.488795] *** VALIDATE nfs4 *** [ 4.494991] pstore: using deflate compression [ 4.507969] Platform Keyring initialized [ 4.969832] NET: Registered protocol family 38 [ 4.971902] Key type asymmetric registered [ 4.974287] Asymmetric key parser 'x509' registered [ 4.978427] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.982705] io scheduler mq-deadline registered [ 4.984822] io scheduler kyber registered [ 4.987323] io scheduler bfq registered [ 4.990456] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.994204] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.998686] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.003527] ACPI: Power Button [PWRF] [ 5.013180] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.021250] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.053992] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.064129] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.104422] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.135721] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.167673] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.179787] Non-volatile memory driver v1.3 [ 5.198724] Linux agpgart interface v0.103 [ 5.375222] virtio_blk virtio1: [vda] 134856 512-byte logical blocks (69.0 MB/65.8 MiB) [ 5.380201] vda: detected capacity change from 0 to 69046272 [ 5.711037] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.714989] vdb: detected capacity change from 0 to 1073741824 [ 5.878440] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.881979] vdc: detected capacity change from 0 to 2621440000 [ 5.902854] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.906329] vdd: detected capacity change from 0 to 2621440000 [ 5.938860] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.949420] vde: detected capacity change from 0 to 4294967296 [ 6.058702] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.084685] vdf: detected capacity change from 0 to 4294967296 [ 6.103674] libphy: Fixed MDIO Bus: probed [ 6.134806] usbcore: registered new interface driver usbserial_generic [ 6.137994] usbserial: USB Serial support registered for generic [ 6.140869] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.146279] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.148370] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.151297] mousedev: PS/2 mouse device common for all mice [ 6.154698] rtc_cmos 00:05: RTC can wake from S4 [ 6.158242] rtc_cmos 00:05: registered as rtc0 [ 6.159960] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.163258] intel_pstate: CPU model not supported [ 6.166149] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.171189] hid: raw HID events driver (C) Jiri Kosina [ 6.171469] usbcore: registered new interface driver usbhid [ 6.174946] usbhid: USB HID core driver [ 6.176158] drop_monitor: Initializing network drop monitor service [ 6.380929] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.381257] Initializing XFRM netlink socket [ 6.385258] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.386397] NET: Registered protocol family 10 [ 6.392043] Segment Routing with IPv6 [ 6.393100] NET: Registered protocol family 17 [ 6.395028] mpls_gso: MPLS GSO support [ 6.401315] RAS: Correctable Errors collector initialized. [ 6.405426] AVX version of gcm_enc/dec engaged. [ 6.407375] AES CTR mode by8 optimization enabled [ 6.633511] sched_clock: Marking stable (6633423429, 0)->(7729986692, -1096563263) [ 6.638936] registered taskstats version 1 [ 6.725925] Loading compiled-in X.509 certificates [ 6.734299] zswap: loaded using pool lzo/zbud [ 6.901628] Key type big_key registered [ 6.911911] Key type encrypted registered [ 6.913292] ima: No TPM chip found, activating TPM-bypass! [ 6.915377] ima: Allocated hash algorithm: sha1 [ 6.917070] ima: No architecture policies found [ 6.918468] evm: Initialising EVM extended attributes: [ 6.920210] evm: security.selinux [ 6.921297] evm: security.ima [ 6.922449] evm: security.capability [ 6.923621] evm: HMAC attrs: 0x1 [ 6.938260] rtc_cmos 00:05: setting system clock to 2026-06-01 16:16:04 UTC (1780330564) [ 6.961161] debug: unmapping init [mem 0xffffffff8b603000-0xffffffff8b7fffff] [ 7.081160] debug: unmapping init [mem 0xffffffff8a382000-0xffffffff8a658fff] [ 7.108460] Write protecting the kernel read-only data: 28672k [ 7.115288] debug: unmapping init [mem 0xffffffff88a03000-0xffffffff88bfffff] [ 7.118554] debug: unmapping init [mem 0xffffffff89314000-0xffffffff893fffff] [ 7.182846] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.200745] systemd[1]: Detected virtualization kvm. [ 7.203932] systemd[1]: Detected architecture x86-64. [ 7.207851] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.284173] systemd[1]: No hostname configured. [ 7.286189] systemd[1]: Set hostname to . [ 7.288784] random: systemd: uninitialized urandom read (16 bytes read) [ 7.294926] systemd[1]: Initializing machine ID from random generator. [ 7.453313] random: ln: uninitialized urandom read (6 bytes read) [ 7.699804] random: systemd: uninitialized urandom read (16 bytes read) [ 7.707557] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 7.715968] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 7.722927] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.546417] device-mapper: uevent: version 1.0.3 [ 10.553628] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [[ 11.931658] random: fast init done  OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ 13.360267] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 17.731245] scsi host0: ata_piix [ 17.747336] scsi host1: ata_piix [ 17.748741] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 17.751053] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 20.860774] random: crng init done [ 20.862630] random: 7 urandom warning(s) missed due to ratelimiting [ 25.451645] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ 30.291346] hrtimer: interrupt took 4986754 ns [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ 30.346309] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 38.241988] printk: systemd: 25 output lines suppressed due to ratelimiting [ 39.428390] SELinux: Disabled at runtime. [ 39.770492] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 39.799897] systemd[1]: Detected virtualization kvm. [ 39.806811] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 44.616637] systemd[1]: initrd-switch-root.service: Succeeded. [ 44.642237] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 44.663773] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 44.679857] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 44.693829] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 44.839838] systemd[1]: Starting Journal Service... Starting Journal Service... [ 45.051912] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 45.109040] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ 45.193852] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 45.291201] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ 45.369931] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 45.575893] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ 45.867792] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ 45.919325] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ 45.979620] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 46.852090] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 50.814308] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 54.750564] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 54.817297] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 55.930551] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 56.116752] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (14s / no limit) [** ] A start job is running for Configur…only root support (15s / no limit) [*** ] A start job is running for Configur…only root support (15s / no limit) [ *** ] A start job is running for Configur…only root support (16s / no limit) [ *** ] A start job is running for Configur…only root support (16s / no limit) [ ***] A start job is running for Configur…only root support (17s / no limit) [ **] A start job is running for Configur…only root support (17s / no limit) [ *] A start job is running for Configur…only root support (18s / no limit) [ **] A start job is running for Configur…only root support (18s / no limit) [ ***] A start job is running for Configur…only root support (19s / no limit) [ *** ] A start job is running for Configur…only root support (19s / no limit) [ *** ] A start job is running for Configur…only root support (20s / no limit) [*** ] A start job is running for Configur…only root support (20s / no limit) [** ] A start job is running for Configur…only root support (21s / no limit) [* ] A start job is running for Configur…only root support (21s / no limit)[ 66.285779] Key type dns_resolver registered [** ] A start job is running for Configur…only root support (22s / no limit) [*** ] A start job is running for Configur…only root support (22s / no limit) [ *** ] A start job is running for Configur…only root support (23s / no limit) [ *** ] A start job is running for Configur…only root support (23s / no limit)[ 67.984916] NFS: Registering the id_resolver key type [ 67.997517] Key type id_resolver registered [ 68.005105] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (24s / no limit) [ **] A start job is running for Configur…only root support (24s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg318-server login: [ 147.201181] libcfs: loading out-of-tree module taints kernel. [ 147.262910] Key type ._llcrypt registered [ 147.264996] Key type .llcrypt registered [ 147.402940] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 164.967393] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 166.455949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 166.480827] alg: No test for adler32 (adler32-zlib) [ 168.160880] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 169.336840] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 171.151180] Key type lgssc registered [ 172.774668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 189.370141] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 230.200744] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 242.965471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 243.046190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 244.404196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 244.444453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 244.569444] Lustre: lustre-MDT0000: new disk, initializing [ 244.715351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 244.735126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 248.875744] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 261.746695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 261.823593] Lustre: 6513:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 261.855845] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 261.864148] Lustre: Skipped 1 previous similar message [ 261.937169] Lustre: lustre-MDT0001: new disk, initializing [ 262.004453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 262.031514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 262.050977] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 266.227080] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 270.804484] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 279.722056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 279.952259] Lustre: lustre-OST0000: new disk, initializing [ 279.956439] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 279.962231] Lustre: 8416:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 280.021403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 286.479243] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 287.264823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 287.278624] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 287.385126] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 300.918297] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 301.101102] Lustre: lustre-OST0001: new disk, initializing [ 301.105304] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 301.110306] Lustre: 9475:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 301.179050] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 307.661859] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 310.315612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 310.327118] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 310.424330] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 320.141771] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 329.962532] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 339.223069] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing check_logdir /tmp/testlogs/ [ 345.433453] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing yml_node [ 350.838208] Lustre: DEBUG MARKER: Client: 2.17.53.30 [ 354.321768] Lustre: DEBUG MARKER: MDS: 2.17.53.30 [ 357.186387] Lustre: DEBUG MARKER: OSS: 2.17.53.30 [ 359.090369] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jun 1 12:21:54 EDT 2026 [ 377.842768] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 379.433705] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 397.791961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 397.793810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 397.807119] Lustre: Skipped 1 previous similar message [ 397.828601] Lustre: Skipped 3 previous similar messages [ 401.805265] Lustre: server umount lustre-MDT0000 complete [ 405.733816] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780330963 with bad export cookie 17810117640002875094 [ 405.735740] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 405.751041] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 408.033098] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 408.044846] LustreError: 6520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 408.055328] Lustre: Skipped 2 previous similar messages [ 408.057037] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 408.114494] LustreError: 6520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 410.081769] LustreError: 8106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 410.082799] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 410.115204] Lustre: Skipped 2 previous similar messages [ 415.203072] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 415.203479] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 415.242091] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 415.263070] Lustre: Skipped 1 previous similar message [ 420.319135] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 420.679243] Lustre: server umount lustre-MDT0001 complete [ 425.109896] Lustre: server umount lustre-OST0000 complete [ 428.734925] Lustre: server umount lustre-OST0001 complete [ 435.376712] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 445.232888] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 482.534972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 482.924735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 482.987278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 483.114547] Lustre: lustre-MDT0000: new disk, initializing [ 483.216381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 483.249273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 487.846904] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 498.944801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 499.095736] Lustre: 14991:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 499.175617] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 499.187378] Lustre: Skipped 1 previous similar message [ 499.317332] Lustre: lustre-MDT0001: new disk, initializing [ 499.472520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 499.543909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 499.559420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 505.157625] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 513.239347] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 516.953191] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 523.437668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 523.782384] Lustre: lustre-OST0000: new disk, initializing [ 523.786094] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 523.792379] Lustre: 16735:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 523.896908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 525.597050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 525.606915] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 525.698452] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 530.590431] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 537.782213] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 543.070346] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 543.423813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 548.288659] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 548.515250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 551.399385] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 551.403461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 551.407653] LustreError: Skipped 1 previous similar message [ 551.412341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 551.427782] Lustre: Skipped 2 previous similar messages [ 551.430396] Lustre: Skipped 2 previous similar messages [ 556.395312] Lustre: server umount lustre-OST0000 complete [ 561.124225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 561.132987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 561.144867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 561.153250] Lustre: Skipped 1 previous similar message [ 566.040148] Lustre: server umount lustre-MDT0000 complete [ 566.752855] LustreError: 15829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.777880] LustreError: 15829:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 569.549458] LustreError: 15780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780331127 with bad export cookie 17810117640002876627 [ 569.556090] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 569.558497] LustreError: 15780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 569.810928] Lustre: server umount lustre-MDT0001 complete [ 573.131128] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 12:25:28 (1780331128) [ 575.198671] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 576.885501] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 12:25:32 (1780331132) [ 582.854435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.218769] LustreError: 18994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.316281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 586.861895] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 592.865582] LustreError: 18995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 592.910634] LustreError: 18995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 595.530759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 599.511494] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 604.302651] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 607.505568] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 613.298841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 618.724682] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 623.526767] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 636.385836] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 636.387573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 636.405197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 636.432957] Lustre: Skipped 2 previous similar messages [ 636.434673] Lustre: Skipped 1 previous similar message [ 642.184853] Lustre: server umount lustre-OST0000 complete [ 646.633542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 646.651311] Lustre: Skipped 1 previous similar message [ 651.698967] Lustre: server umount lustre-MDT0000 complete [ 651.745297] LustreError: 18990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 651.763365] LustreError: 18990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 655.575037] LustreError: 18975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780331213 with bad export cookie 17810117640002877509 [ 655.583267] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 655.588575] LustreError: 18975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 655.909739] Lustre: server umount lustre-MDT0001 complete [ 666.233359] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 668.741545] Key type lgssc unregistered [ 669.058872] LNet: 22188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 669.078257] LNetError: 22188:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 670.126765] LNet: Removed LNI 192.168.203.118@tcp [ 670.973190] Key type .llcrypt unregistered [ 670.976430] Key type ._llcrypt unregistered [ 689.597366] Key type ._llcrypt registered [ 689.602313] Key type .llcrypt registered [ 689.718205] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 690.552964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 690.710213] alg: No test for adler32 (adler32-zlib) [ 691.711357] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 691.932910] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 693.591560] Key type lgssc registered [ 694.634310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 701.413965] loop: module loaded [ 701.468410] loop0: detected capacity change from 0 to 204800000 [ 701.661293] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 701.669470] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 701.914431] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 707.383715] loop0: detected capacity change from 0 to 204800000 [ 707.594515] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 707.610134] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 707.635678] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 712.802426] loop0: detected capacity change from 0 to 204800000 [ 718.204134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 718.226427] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 719.500748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 719.577617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 719.706806] Lustre: lustre-MDT0000: new disk, initializing [ 719.808450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 719.827753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 723.527146] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 731.201835] loop1: detected capacity change from 0 to 204800000 [ 735.854680] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 736.010068] Lustre: lustre-OST07c6: new disk, initializing [ 736.013093] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 736.018935] Lustre: Skipped 1 previous similar message [ 736.024171] Lustre: 25321:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 736.106888] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 742.194118] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 746.094937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 746.111193] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 746.172839] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 747.827572] Lustre: 25820:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 747.846096] Lustre: 25820:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 747.853203] Lustre: 25820:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 747.874587] Lustre: 25820:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 747.884812] Lustre: 25820:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 747.896573] Lustre: 25820:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 760.987051] Lustre: 24323:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 761.009458] Lustre: 24323:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 761.022320] Lustre: 24323:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 761.025314] Lustre: 24323:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 761.028692] Lustre: 24323:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 761.031705] Lustre: 24323:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 765.774421] Lustre: 24323:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 765.781075] Lustre: 24323:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 765.788813] Lustre: 24323:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 765.792699] Lustre: 24323:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.796825] Lustre: 24323:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 765.801772] Lustre: 24323:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.809036] Lustre: 24323:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 765.813994] Lustre: 24323:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.819757] Lustre: 24323:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 765.822755] Lustre: 24323:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.826718] Lustre: 24323:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 765.830206] Lustre: 24323:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.004866] Lustre: 24325:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 769.011037] Lustre: 24325:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 769.014989] Lustre: 24325:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 769.021483] Lustre: 24325:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.029033] Lustre: 24325:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 769.037939] Lustre: 24325:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.043787] Lustre: 24325:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 769.050241] Lustre: 24325:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.058300] Lustre: 24325:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 769.064922] Lustre: 24325:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.075289] Lustre: 24325:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 769.083505] Lustre: 24325:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.373720] Lustre: 24325:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 773.404607] Lustre: 24325:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 773.415586] Lustre: 24325:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 773.422838] Lustre: 24325:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.429930] Lustre: 24325:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 773.438722] Lustre: 24325:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.450759] Lustre: 24325:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 773.465384] Lustre: 24325:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.478051] Lustre: 24325:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 773.487232] Lustre: 24325:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.499960] Lustre: 24325:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 773.517034] Lustre: 24325:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 784.753379] Lustre: 25820:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 784.768467] Lustre: 25820:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 784.779132] Lustre: 25820:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 784.784420] Lustre: 25820:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 784.795334] Lustre: 25820:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 784.801064] Lustre: 25820:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 784.806423] Lustre: 25820:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 784.815314] Lustre: 25820:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 784.821646] Lustre: 25820:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 784.827374] Lustre: 25820:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 784.834555] Lustre: 25820:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 784.840226] Lustre: 25820:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 832.992167] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 833.018731] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 837.784772] Lustre: server umount lustre-OST07c6 complete [ 847.062711] Lustre: server umount lustre-MDT0000 complete [ 861.440209] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 864.277986] Key type lgssc unregistered [ 864.614737] LNet: 27864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.622063] LNetError: 27864:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.648329] LNet: Removed LNI 192.168.203.118@tcp [ 865.383198] Key type .llcrypt unregistered [ 865.386441] Key type ._llcrypt unregistered [ 880.039535] Key type ._llcrypt registered [ 880.041783] Key type .llcrypt registered [ 880.125695] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 880.732477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 880.800109] alg: No test for adler32 (adler32-zlib) [ 881.866421] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 882.071717] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 883.767175] Key type lgssc registered [ 884.869163] Lustre: Echo OBD driver; http://www.lustre.org/ [ 892.916918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 892.939430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 894.562865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 898.649421] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 904.674370] LustreError: 29098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 904.696745] LustreError: 29098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 905.799732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 906.060332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 909.249187] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 913.890199] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 917.467942] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 922.796508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 923.054464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 927.962973] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 933.521530] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 940.465990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 955.873201] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 955.882626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 955.889315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 955.901464] Lustre: Skipped 1 previous similar message [ 955.909335] Lustre: Skipped 1 previous similar message [ 960.997625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 961.008013] Lustre: Skipped 1 previous similar message [ 961.292545] Lustre: server umount lustre-OST0000 complete [ 966.113096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 966.131496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 966.138547] Lustre: Skipped 1 previous similar message [ 970.640523] Lustre: server umount lustre-MDT0000 complete [ 971.233401] LustreError: 29098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 971.250744] LustreError: 29098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 974.327617] LustreError: 29078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780331531 with bad export cookie 11274301581788822152 [ 974.342307] LustreError: 29078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 974.342922] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 974.673270] Lustre: server umount lustre-MDT0001 complete [ 984.318514] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 986.816871] Key type lgssc unregistered [ 987.045849] LNet: 32497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.056377] LNetError: 32497:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.078073] LNet: Removed LNI 192.168.203.118@tcp [ 987.750148] Key type .llcrypt unregistered [ 987.755487] Key type ._llcrypt unregistered [ 1003.351951] Key type ._llcrypt registered [ 1003.356253] Key type .llcrypt registered [ 1003.445515] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 1004.130565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1004.217279] alg: No test for adler32 (adler32-zlib) [ 1005.254923] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 1005.464307] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 1007.103208] Key type lgssc registered [ 1008.166721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1014.510699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1014.537176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1015.977598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1019.557474] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1025.511099] LustreError: 33743:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1025.531843] LustreError: 33743:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1026.810790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1027.044628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1031.456207] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1036.756888] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1040.073781] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1046.861862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1047.155407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1052.524670] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1057.109711] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1060.464973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1073.120889] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1073.129711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1073.144461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1076.197867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1076.201659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1076.208169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1077.535274] Lustre: server umount lustre-OST0000 complete [ 1081.312862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1081.317438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1081.329019] Lustre: Skipped 1 previous similar message [ 1081.344407] Lustre: Skipped 2 previous similar messages [ 1086.433320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1086.441434] Lustre: Skipped 1 previous similar message [ 1091.554974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1091.565337] Lustre: Skipped 1 previous similar message [ 1095.135131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1095.483302] Lustre: server umount lustre-MDT0000 complete [ 1096.678664] LustreError: 33738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1096.691279] LustreError: 33738:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1099.165673] LustreError: 33722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780331656 with bad export cookie 15950205235439434094 [ 1099.177368] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1099.180273] LustreError: 33722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1099.487944] Lustre: server umount lustre-MDT0001 complete [ 1110.340277] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 1113.085078] Key type lgssc unregistered [ 1113.326660] LNet: 36942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1113.336148] LNetError: 36942:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1113.359688] LNet: Removed LNI 192.168.203.118@tcp [ 1114.070714] Key type .llcrypt unregistered [ 1114.075352] Key type ._llcrypt unregistered [ 1132.899287] Key type ._llcrypt registered [ 1132.902164] Key type .llcrypt registered [ 1133.011736] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 1133.820133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1133.889703] alg: No test for adler32 (adler32-zlib) [ 1134.914912] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 1135.104313] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 1136.761156] Key type lgssc registered [ 1137.692839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1144.460284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1144.479725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1146.046971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1150.030456] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1155.557450] LustreError: 38319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1155.574919] LustreError: 38319:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1157.298270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1157.585437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1161.133658] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1166.129720] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1169.165469] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1175.505957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1175.928234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1181.740247] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1186.729727] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1191.279037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1196.513471] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.515509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1196.536291] Lustre: Skipped 1 previous similar message [ 1197.636215] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.18@tcp (stopping) [ 1197.644811] Lustre: Skipped 2 previous similar messages [ 1198.733662] Lustre: server umount lustre-MDT0000 complete [ 1201.633891] LustreError: 38320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1201.667464] LustreError: 38320:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1202.362136] LustreError: 38303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780331759 with bad export cookie 10338833787210451772 [ 1202.370347] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1202.373382] LustreError: 38303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1202.743983] LustreError: 38314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1202.780112] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.18@tcp (stopping) [ 1206.752781] LustreError: 38315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1206.756664] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1206.795022] Lustre: Skipped 1 previous similar message [ 1206.803433] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1208.755316] Lustre: server umount lustre-MDT0001 complete [ 1229.488771] Lustre: server umount lustre-OST0000 complete [ 1244.112351] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 1246.822837] Key type lgssc unregistered [ 1247.125074] LNet: 41821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1247.132481] LNetError: 41821:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1247.150054] LNet: Removed LNI 192.168.203.118@tcp [ 1247.911626] Key type .llcrypt unregistered [ 1247.914776] Key type ._llcrypt unregistered [ 1270.386227] Key type ._llcrypt registered [ 1270.390116] Key type .llcrypt registered [ 1270.500264] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 1271.350102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1271.405017] alg: No test for adler32 (adler32-zlib) [ 1272.589646] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 1272.872966] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 1274.551368] Key type lgssc registered [ 1275.539568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1282.689649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1282.719284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1284.201243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1287.720992] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1293.799460] LustreError: 43236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1293.814673] LustreError: 43236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1295.830406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1296.179592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1300.056840] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1305.328942] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1309.846396] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1316.967646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1317.520253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1324.133893] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1328.817421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1329.629538] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1336.290204] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1336.301163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.318909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1337.425305] Lustre: lustre-OST0000: Not available for connect from 192.168.203.18@tcp (stopping) [ 1339.364513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1339.366284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1339.376399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1339.397546] Lustre: Skipped 1 previous similar message [ 1341.795871] Lustre: server umount lustre-OST0000 complete [ 1359.856209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1359.866669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1359.869513] Lustre: Skipped 1 previous similar message [ 1359.890218] Lustre: Skipped 1 previous similar message [ 1364.706363] Lustre: server umount lustre-MDT0000 complete [ 1364.964828] LustreError: 43233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1364.985508] LustreError: 43233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1365.987438] LustreError: 43236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1366.000962] LustreError: 43236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1367.737077] LustreError: 45597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780331925 with bad export cookie 16691459684045564617 [ 1367.743394] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1367.750215] LustreError: 45597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1382.367169] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1382.617791] Lustre: server umount lustre-MDT0001 complete [ 1391.102934] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 1393.458150] Key type lgssc unregistered [ 1393.687763] LNet: 46589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1393.694149] LNetError: 46589:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1393.722397] LNet: Removed LNI 192.168.203.118@tcp [ 1394.305619] Key type .llcrypt unregistered [ 1394.307807] Key type ._llcrypt unregistered [ 1412.445205] Key type ._llcrypt registered [ 1412.448455] Key type .llcrypt registered [ 1412.557930] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 1413.317199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1413.328859] alg: No test for adler32 (adler32-zlib) [ 1414.287187] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 1414.535247] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 1416.207174] Key type lgssc registered [ 1417.102525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1423.933742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1423.945940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1425.370965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1430.048095] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1434.597576] LustreError: 47968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1434.611713] LustreError: 47968:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1437.186864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1437.542442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1441.419936] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1446.293696] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1449.029241] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1454.330842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1454.601048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1459.711329] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1464.308531] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1470.014589] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1478.545946] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1481.029485] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1492.884548] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-0112h40m47s [ 1493.968575] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1495.521366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1495.523020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1495.537609] Lustre: Skipped 2 previous similar messages [ 1495.546096] Lustre: Skipped 2 previous similar messages [ 1496.611642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1496.619571] Lustre: Skipped 3 previous similar messages [ 1501.624754] Lustre: server umount lustre-MDT0000 complete [ 1501.733161] LustreError: 47962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1504.348364] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1507.879561] LustreError: 49174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1507.894794] LustreError: 49174:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1509.681639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1509.784067] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1510.037907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1510.088051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1511.086555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1514.368959] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1517.498541] Lustre: DEBUG MARKER: Wait for df (36004) ... [ 1518.651083] Lustre: DEBUG MARKER: done [ 1524.194269] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1524.207934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1524.228513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1524.236534] Lustre: Skipped 1 previous similar message [ 1525.731268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1525.753431] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1526.764509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1526.790254] Lustre: Skipped 2 previous similar messages [ 1528.309987] Lustre: server umount lustre-OST0000 complete [ 1531.874251] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1531.879885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1531.895556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1531.904288] Lustre: Skipped 1 previous similar message [ 1531.920786] Lustre: Skipped 1 previous similar message [ 1544.160343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1544.166141] Lustre: Skipped 5 previous similar messages [ 1546.207131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1546.337030] Lustre: server umount lustre-MDT0000 complete [ 1549.630375] LustreError: 50276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780332107 with bad export cookie 10284214963056287563 [ 1549.637320] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1549.647847] LustreError: 50276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1550.190311] Lustre: server umount lustre-MDT0001 complete [ 1560.041844] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 1562.278970] Key type lgssc unregistered [ 1562.500717] LNet: 52538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1562.506803] LNetError: 52538:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1562.522204] LNet: Removed LNI 192.168.203.118@tcp [ 1563.494909] Key type .llcrypt unregistered [ 1563.499101] Key type ._llcrypt unregistered [ 1603.979564] Key type ._llcrypt registered [ 1603.981445] Key type .llcrypt registered [ 1604.199675] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 1605.749701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1605.841040] alg: No test for adler32 (adler32-zlib) [ 1607.216920] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 1607.522889] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 1609.272080] Key type lgssc registered [ 1610.671959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1619.341082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1619.384755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1620.776113] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1620.821403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1620.956384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1624.407868] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1631.655931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1631.726022] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1631.753585] Lustre: 54733:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1632.024183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1635.680526] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1641.024324] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1644.609719] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1650.819185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1651.007284] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1651.216075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1656.110489] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1659.548636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1661.094718] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1666.421555] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-0112h43m41s [ 1668.171441] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1676.088662] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1679.066766] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1679.143051] LustreError: 54738:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1699.398943] Lustre: lustre-MDT0000: Client bda8df2a-c886-4012-86bb-a719e59e7c15 (at 192.168.203.18@tcp) reconnecting [ 1719.207171] LustreError: 54738:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1720.975060] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1722.901699] Lustre: DEBUG MARKER: done [ 1731.042533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1731.053587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1731.072534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1731.556758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1731.570057] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1733.147870] Lustre: server umount lustre-OST0000 complete [ 1736.675608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1736.677716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1736.684619] Lustre: Skipped 1 previous similar message [ 1736.691988] Lustre: Skipped 1 previous similar message [ 1737.761340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1737.768567] Lustre: Skipped 1 previous similar message [ 1742.019830] Lustre: server umount lustre-MDT0000 complete [ 1744.938751] LustreError: 54724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780332302 with bad export cookie 12082573055518504581 [ 1744.951635] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1744.953276] LustreError: 54724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1745.233320] Lustre: server umount lustre-MDT0001 complete [ 1753.821186] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 1756.220234] Key type lgssc unregistered [ 1756.454600] LNet: 58698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1756.462252] LNetError: 58698:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1756.471568] LNet: Removed LNI 192.168.203.118@tcp [ 1757.323175] Key type .llcrypt unregistered [ 1757.327408] Key type ._llcrypt unregistered [ 1795.171490] Key type ._llcrypt registered [ 1795.173986] Key type .llcrypt registered [ 1795.384512] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 1797.337411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1797.403081] alg: No test for adler32 (adler32-zlib) [ 1798.562805] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 1798.811220] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 1800.543155] Key type lgssc registered [ 1801.986712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1806.077219] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1806.108323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1807.381935] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1807.420752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1807.658169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1808.811860] Lustre: Failing over lustre-MDT0000 [ 1809.179946] Lustre: server umount lustre-MDT0000 complete [ 1815.633977] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 12:46:10 (1780332370) [ 1822.085452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1822.651457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1827.221905] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1835.671773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1835.802068] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1835.842391] Lustre: 61481:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1836.186656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1840.626242] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1846.218617] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1850.359986] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1856.422931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1856.651866] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1856.914289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1862.405452] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1864.355100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1867.395921] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1873.323234] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1882.082438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1882.091479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.103879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1885.152415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1885.157078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1885.162382] Lustre: Skipped 1 previous similar message [ 1885.191506] Lustre: Skipped 1 previous similar message [ 1886.513398] Lustre: server umount lustre-MDT0000 complete [ 1889.575037] LustreError: 63107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780332447 with bad export cookie 15271804017446559549 [ 1889.580351] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1889.583214] LustreError: 63107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1889.880549] Lustre: server umount lustre-MDT0001 complete [ 1892.900981] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1896.938920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1904.752283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1905.631344] Lustre: 60404:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780332447/real 1780332447] req@0000000078b4aa67 x1866813780564352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780332463 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1905.663913] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1909.729112] Lustre: 60405:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780332451/real 1780332451] req@0000000031d1cc4f x1866813780564480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780332467 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1910.751515] Lustre: 60403:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780332452/real 1780332452] req@000000004eded5ae x1866813780564608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780332468 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1914.912855] LustreError: 60401:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003346c51f x1866813780565376/t0(0) o250->MGC192.168.203.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1915.398530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1915.501761] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1916.386324] Lustre: 60403:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780332457/real 1780332457] req@0000000076c02272 x1866813780564864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780332473 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1918.935617] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1920.864120] Lustre: 60405:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780332462/real 1780332462] req@00000000bf371560 x1866813780564992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780332478 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1921.507160] LustreError: 64910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1921.544323] LustreError: 64910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1924.579994] LustreError: 64908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1924.583101] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1924.595233] LustreError: 64908:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1927.231451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1927.586229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1931.916969] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1932.774257] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1937.228817] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1941.146895] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1948.129758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.133284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1948.149770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1948.158026] Lustre: Skipped 3 previous similar messages [ 1953.248910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1953.262677] Lustre: Skipped 2 previous similar messages [ 1953.956623] Lustre: server umount lustre-MDT0000 complete [ 1957.472215] LustreError: 61473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780332515 with bad export cookie 15271804017446561824 [ 1957.491548] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1957.923443] Lustre: server umount lustre-MDT0001 complete [ 1964.072355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1973.031772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.727211] Lustre: 60404:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780332515/real 1780332515] req@00000000459e2c77 x1866813780589440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780332531 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1973.763448] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.780139] Lustre: Skipped 2 previous similar messages [ 1983.007792] LustreError: 60401:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f9a6ce88 x1866813780590464/t0(0) o250->MGC192.168.203.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1983.316357] LustreError: 63101:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1983.384988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1983.441802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1987.452184] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1989.600791] LustreError: 67364:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1989.623119] LustreError: 67364:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1994.721125] LustreError: 67366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1994.739629] LustreError: 67366:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1996.046202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2000.533445] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2001.396681] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2006.578219] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2011.206510] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2021.863098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2021.870856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2021.891422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2024.828191] Lustre: server umount lustre-MDT0000 complete [ 2028.011474] LustreError: 67741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2028.037383] LustreError: 67741:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2028.862280] LustreError: 61473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780332586 with bad export cookie 15271804017446563203 [ 2028.864881] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2028.872108] LustreError: 61473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2029.270590] Lustre: server umount lustre-MDT0001 complete [ 2036.313989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2041.066158] Lustre: DEBUG MARKER: files compared the same [ 2053.553816] Lustre: server umount lustre-OST0000 complete [ 2071.903379] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 2075.544740] Key type lgssc unregistered [ 2075.809652] LNet: 70398:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2075.817584] LNetError: 70398:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2075.838617] LNet: Removed LNI 192.168.203.118@tcp [ 2076.659914] Key type .llcrypt unregistered [ 2076.663300] Key type ._llcrypt unregistered [ 2102.633305] Key type ._llcrypt registered [ 2102.637643] Key type .llcrypt registered [ 2102.832467] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 2104.252360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2104.286191] alg: No test for adler32 (adler32-zlib) [ 2105.359497] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 2105.666190] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 2107.471658] Key type lgssc registered [ 2108.568937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2115.959607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2115.973448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.549805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2122.117102] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug +malloc all [ 2127.850148] LustreError: 71800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2127.874732] LustreError: 71800:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2130.702465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.995645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2135.487379] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug +malloc all [ 2140.891481] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2144.481581] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2151.748887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2152.098593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2158.834743] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug +malloc all [ 2163.375306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2165.340971] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2176.998949] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2177.022042] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.055833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2179.043968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2179.045569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2179.067775] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2180.791516] Lustre: server umount lustre-OST0000 complete [ 2187.232551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2187.238978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2187.249189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2187.252266] Lustre: Skipped 1 previous similar message [ 2189.280793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2189.281121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2191.227899] Lustre: server umount lustre-MDT0000 complete [ 2194.409155] LustreError: 71794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.435388] LustreError: 71794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2195.265919] LustreError: 71780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780332752 with bad export cookie 11451180188661280886 [ 2195.269444] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2195.278233] LustreError: 71780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2195.524910] Lustre: server umount lustre-MDT0001 complete [ 2206.307353] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 2209.562510] Key type lgssc unregistered [ 2209.966475] LNet: 74975:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2209.978468] LNetError: 74975:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2209.990673] LNet: Removed LNI 192.168.203.118@tcp [ 2211.144210] Key type .llcrypt unregistered [ 2211.150338] Key type ._llcrypt unregistered [ 2236.805363] Key type ._llcrypt registered [ 2236.809031] Key type .llcrypt registered [ 2236.931078] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 2237.994032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2238.022627] alg: No test for adler32 (adler32-zlib) [ 2239.120872] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 2239.374286] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 2241.071163] Key type lgssc registered [ 2242.256641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2250.308967] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2250.332206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2266.015360] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2271.135258] LustreError: 76346:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.118@tcp: failed processing log, type 4: rc = -110 [ 2297.824577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2305.149200] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2311.385227] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2320.100122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2320.272184] Lustre: *** cfs_fail_loc=706, val=0*** [ 2320.762262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2320.828638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2325.626775] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2325.985195] LustreError: 76371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2326.012151] LustreError: 76371:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2331.112416] LustreError: 76372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.163888] LustreError: 77321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.196738] LustreError: 77321:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2334.988984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2335.471267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2339.873415] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2345.459576] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2349.531296] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2351.586579] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2351.595386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2351.611913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2356.194737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2356.197942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2356.210693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2357.745206] Lustre: server umount lustre-OST0000 complete [ 2361.314935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2361.329251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2361.335249] Lustre: Skipped 2 previous similar messages [ 2366.433291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2366.447884] Lustre: Skipped 1 previous similar message [ 2367.722482] Lustre: server umount lustre-MDT0000 complete [ 2371.347082] LustreError: 76353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780332928 with bad export cookie 6858070622984015936 [ 2371.355369] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2371.362272] LustreError: 76353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2371.777609] Lustre: server umount lustre-MDT0001 complete [ 2382.522187] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 2385.615634] Key type lgssc unregistered [ 2385.915445] LNet: 79606:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2385.922736] LNetError: 79606:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2385.946367] LNet: Removed LNI 192.168.203.118@tcp [ 2386.785450] Key type .llcrypt unregistered [ 2386.787197] Key type ._llcrypt unregistered [ 2410.375678] Key type ._llcrypt registered [ 2410.379530] Key type .llcrypt registered [ 2410.475106] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 2411.207372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2411.252116] alg: No test for adler32 (adler32-zlib) [ 2412.221670] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 2412.493956] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 2414.199229] Key type lgssc registered [ 2415.252428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2423.226562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2423.255692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.014911] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2435.392765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2435.816361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2439.812757] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2441.186535] LustreError: 81679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2441.208695] LustreError: 81679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2446.313327] LustreError: 81678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.329746] LustreError: 81678:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2446.862499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2447.249950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2452.461942] LustreError: 81682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.488826] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2452.489581] LustreError: 81682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2457.571495] LustreError: 81683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2457.594797] LustreError: 81683:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2461.448480] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2461.459248] Lustre: MGS: Client f09b2216-07ff-46ad-b400-936b6edf536f (at 0@lo) reconnecting [ 2461.491369] LustreError: 81693:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2461.862132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2461.961296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2467.573994] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2482.149651] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2482.157823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2482.167927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2487.789454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2487.810305] Lustre: Skipped 2 previous similar messages [ 2488.497531] Lustre: server umount lustre-OST0000 complete [ 2492.898771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2492.904074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2492.923271] Lustre: Skipped 1 previous similar message [ 2492.924742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2498.017337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2498.027240] Lustre: Skipped 2 previous similar messages [ 2498.814861] Lustre: server umount lustre-MDT0000 complete [ 2502.787593] Lustre: server umount lustre-MDT0001 complete [ 2506.428652] Lustre: server umount MGS complete [ 2517.864740] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 12:57:53 (1780333073) [ 2549.897387] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 2560.712758] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 2597.700968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2602.963240] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2612.614508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2612.927729] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2612.979185] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2613.086597] Lustre: lustre-MDT0001: new disk, initializing [ 2613.156365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2613.175341] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2613.184146] Lustre: 88766:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2613.189267] LustreError: 88766:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2617.493336] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2627.587646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2627.867783] Lustre: lustre-OST0000: new disk, initializing [ 2627.872746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2627.879659] Lustre: 89515:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2627.960623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2634.303176] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2638.432197] Lustre: 89960:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000006e8aa9da x1866814424391168/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1780333211 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2641.746083] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2648.038233] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2648.048054] Lustre: MGS: Client 0903e172-0b81-4d7f-b5de-666421cc9216 (at 0@lo) reconnecting [ 2648.094478] Lustre: 90448:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2648.112608] LustreError: 88752:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2648.125996] LustreError: 88752:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2648.152871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2648.243434] Lustre: lustre-MDT0000: new disk, initializing [ 2648.345780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2648.363621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2648.441946] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2649.440448] Lustre: 90473:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000069fb920d x1866814424397696/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1780333223 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2652.245907] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2658.278220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2658.290199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2658.620257] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2683.873590] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2683.884871] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2683.898870] Lustre: Skipped 1 previous similar message [ 2683.903923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2688.259343] Lustre: server umount lustre-OST0000 complete [ 2696.673805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2696.678520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2696.705124] Lustre: Skipped 1 previous similar message [ 2696.709763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2696.720995] LustreError: Skipped 1 previous similar message [ 2696.745700] Lustre: Skipped 4 previous similar messages [ 2698.178802] Lustre: server umount lustre-MDT0000 complete [ 2701.793840] LustreError: 88742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2701.820117] LustreError: 88742:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2708.245299] Lustre: server umount lustre-MDT0001 complete [ 2713.614468] Lustre: server umount MGS complete [ 2727.357450] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 13:01:22 (1780333282) [ 2745.651796] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 2748.552741] Key type lgssc unregistered [ 2748.822417] LNet: 92949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2748.831648] LNetError: 92949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2748.843743] LNet: Removed LNI 192.168.203.118@tcp [ 2749.603936] Key type .llcrypt unregistered [ 2749.605939] Key type ._llcrypt unregistered [ 2766.430839] Key type ._llcrypt registered [ 2766.436306] Key type .llcrypt registered [ 2766.571913] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 2767.828385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2767.868185] alg: No test for adler32 (adler32-zlib) [ 2769.119938] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 2769.342038] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 2771.055254] Key type lgssc registered [ 2772.260937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2780.746410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2780.775790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2782.293804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2786.502816] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2792.417184] LustreError: 94153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2792.452060] LustreError: 94153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2794.593779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.976819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2800.011684] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2805.729969] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2809.319467] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2816.966876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2817.487788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2823.962834] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2826.679279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2830.349771] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2840.035891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2840.045352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2840.064130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2842.080891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2842.087778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2842.093834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2842.112020] Lustre: Skipped 1 previous similar message [ 2844.727493] Lustre: server umount lustre-OST0000 complete [ 2851.296473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2851.305468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2851.322653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2854.800738] Lustre: server umount lustre-MDT0000 complete [ 2857.445498] LustreError: 94149:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2857.460284] LustreError: 94149:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2858.474626] LustreError: 94132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780333416 with bad export cookie 10365088039754611701 [ 2858.480191] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2858.485235] LustreError: 94132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2858.901279] Lustre: server umount lustre-MDT0001 complete [ 2869.677993] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 2871.846589] Key type lgssc unregistered [ 2872.088921] LNet: 97331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2872.099286] LNetError: 97331:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2872.121484] LNet: Removed LNI 192.168.203.118@tcp [ 2872.856206] Key type .llcrypt unregistered [ 2872.859498] Key type ._llcrypt unregistered [ 2886.672940] Key type ._llcrypt registered [ 2886.675270] Key type .llcrypt registered [ 2886.788310] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 2888.040811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2888.068887] alg: No test for adler32 (adler32-zlib) [ 2889.181765] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 2889.410558] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 2891.096189] Key type lgssc registered [ 2892.060827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2897.419034] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2897.428496] LustreError: 98373:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 2897.454330] LustreError: 98377:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 2897.460472] LustreError: 98373:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4977 [ 2897.467410] LustreError: 98373:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2897.479172] LustreError: 98373:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2897.503237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2898.917351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2905.869960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2906.193124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2910.087610] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2915.627237] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing lsmod [ 2918.747201] LustreError: 99542:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 2918.823290] LustreError: 99549:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 2918.829722] LustreError: 99542:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4926 [ 2918.837116] LustreError: 99542:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2918.850516] LustreError: 99542:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2918.859980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2919.227570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2924.512752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2924.517028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2924.527113] Lustre: Skipped 1 previous similar message [ 2924.538127] Lustre: Skipped 2 previous similar messages [ 2927.972906] Lustre: server umount lustre-MDT0000 complete [ 2929.635667] LustreError: 98415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2929.652784] LustreError: 98415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2931.627838] LustreError: 98391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780333489 with bad export cookie 5574295743753713048 [ 2931.641624] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2931.656849] LustreError: 98411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2931.662210] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2931.718477] Lustre: Skipped 1 previous similar message [ 2931.733117] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2932.144403] Lustre: server umount lustre-MDT0001 complete [ 2935.963154] Lustre: server umount lustre-OST0000 complete [ 2945.517381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2945.942143] LustreError: 100612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2946.048492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2951.030686] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2951.137781] LustreError: 100613:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2951.159706] LustreError: 100613:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2956.263555] LustreError: 100612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2960.117729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2960.582549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2964.667493] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2969.941608] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2973.926373] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2980.164258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2980.416539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2985.712581] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2990.612413] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2990.637337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3000.801606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3000.804105] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3000.818397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3000.834471] Lustre: Skipped 1 previous similar message [ 3000.843644] Lustre: Skipped 1 previous similar message [ 3002.928521] Lustre: server umount lustre-OST0000 complete [ 3006.437055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3006.443936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3006.459052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3011.040832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3011.041157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3011.061267] Lustre: Skipped 1 previous similar message [ 3012.718368] Lustre: server umount lustre-MDT0000 complete [ 3015.932798] LustreError: 100592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780333573 with bad export cookie 5574295743753713727 [ 3015.947518] LustreError: 100592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3015.948799] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3016.242974] Lustre: server umount lustre-MDT0001 complete [ 3028.048253] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 3031.356380] Key type lgssc unregistered [ 3031.700856] LNet: 103787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3031.708319] LNetError: 103787:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3031.730539] LNet: Removed LNI 192.168.203.118@tcp [ 3032.642666] Key type .llcrypt unregistered [ 3032.645758] Key type ._llcrypt unregistered [ 3057.984131] Key type ._llcrypt registered [ 3057.989769] Key type .llcrypt registered [ 3058.180697] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 3059.014796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3059.101105] alg: No test for adler32 (adler32-zlib) [ 3060.092337] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 3060.306301] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 3061.999125] Key type lgssc registered [ 3063.409274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3071.404376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3071.458657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3073.161966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3078.008389] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3082.722462] LustreError: 105189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3082.747556] LustreError: 105189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3086.735153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3087.176412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3091.726898] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3097.721704] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3101.437385] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3109.917053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3110.600098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3117.783561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3119.715943] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3126.774557] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3132.876377] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3135.931079] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3142.634034] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3142.649051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3142.666455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3143.661227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3143.665211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3146.542123] Lustre: server umount lustre-OST0000 complete [ 3153.894296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3153.896748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3153.897357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3153.897361] Lustre: Skipped 1 previous similar message [ 3153.921978] Lustre: Skipped 1 previous similar message [ 3157.755204] Lustre: server umount lustre-MDT0000 complete [ 3159.021906] LustreError: 105185:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3159.050070] LustreError: 105185:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3161.782254] LustreError: 105170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780333719 with bad export cookie 5610202960632736868 [ 3161.800783] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3161.807027] LustreError: 105170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3162.296483] Lustre: server umount lustre-MDT0001 complete [ 3174.205285] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 3177.007335] Key type lgssc unregistered [ 3177.264029] LNet: 108473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3177.269381] LNetError: 108473:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3178.345722] LNet: Removed LNI 192.168.203.118@tcp [ 3179.114579] Key type .llcrypt unregistered [ 3179.118529] Key type ._llcrypt unregistered [ 3195.200366] Key type ._llcrypt registered [ 3195.204905] Key type .llcrypt registered [ 3195.361536] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 3196.228964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3196.342685] alg: No test for adler32 (adler32-zlib) [ 3197.443143] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 3197.736268] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 3199.431176] Key type lgssc registered [ 3200.410909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3207.584768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3207.627823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3209.113093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3213.542508] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3219.424977] LustreError: 109678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.437607] LustreError: 109678:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3221.302272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3221.644952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3226.088025] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3231.405946] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3235.015813] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3242.422161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3242.725936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3248.514407] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3253.794557] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3254.001649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3262.432375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3262.444681] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3262.461464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3264.481740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3264.483550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3267.670736] Lustre: server umount lustre-OST0000 complete [ 3272.672445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3272.691315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3272.722642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3272.733671] Lustre: Skipped 1 previous similar message [ 3274.724330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3274.727357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3274.750431] Lustre: Skipped 1 previous similar message [ 3277.851915] Lustre: server umount lustre-MDT0000 complete [ 3279.846567] LustreError: 111826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3279.878293] LustreError: 111826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3281.539283] LustreError: 109660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780333839 with bad export cookie 12504472090557692609 [ 3281.541164] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3281.548780] LustreError: 109660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3281.847273] Lustre: server umount lustre-MDT0001 complete [ 3292.193128] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 3295.103880] Key type lgssc unregistered [ 3295.387844] LNet: 112853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3295.397626] LNetError: 112853:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3295.444267] LNet: Removed LNI 192.168.203.118@tcp [ 3296.289935] Key type .llcrypt unregistered [ 3296.293549] Key type ._llcrypt unregistered [ 3323.146026] Key type ._llcrypt registered [ 3323.148567] Key type .llcrypt registered [ 3323.245627] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 3324.302812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3324.457681] alg: No test for adler32 (adler32-zlib) [ 3325.671929] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 3325.937092] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 3327.607623] Key type lgssc registered [ 3328.549851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3335.963573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3335.977694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3337.462284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3341.728328] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3346.919097] LustreError: 114254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3346.950663] LustreError: 114254:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3352.033223] LustreError: 114255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3352.066073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3352.520727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3357.044805] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3362.150476] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3365.912699] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3373.694224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3374.028629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3380.102664] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3382.276454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3385.362286] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3392.675867] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3396.227282] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3396.240744] Lustre: Skipped 2 previous similar messages [ 3398.730788] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3398.784258] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3400.369509] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3400.381139] Lustre: Skipped 2 previous similar messages [ 3405.897242] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3405.900609] Lustre: Skipped 1 previous similar message [ 3413.401757] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.18@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3413.416559] Lustre: Skipped 2 previous similar messages [ 3413.592651] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.18@tcp 192.168.0.[2,10]@tcp [ 3417.763086] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.18/24@tcp in log lustre-MDT0000 [ 3417.772553] Lustre: Skipped 2 previous similar messages [ 3420.529068] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.18/24@tcp [ 3420.549220] Lustre: Skipped 1 previous similar message [ 3428.342480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3428.342930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3428.360871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3428.362783] LustreError: Skipped 1 previous similar message [ 3428.371297] Lustre: Skipped 1 previous similar message [ 3433.458648] LustreError: 115850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3433.477498] LustreError: 115850:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3433.632595] Lustre: server umount lustre-OST0000 complete [ 3438.564869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3438.573444] Lustre: Skipped 1 previous similar message [ 3438.577954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3443.680440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3443.687444] Lustre: Skipped 2 previous similar messages [ 3448.801098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3448.809559] Lustre: Skipped 1 previous similar message [ 3452.383760] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3452.687540] Lustre: server umount lustre-MDT0000 complete [ 3453.921046] LustreError: 114255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3453.972705] LustreError: 114255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3457.450533] LustreError: 114972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334015 with bad export cookie 12242457320857723368 [ 3457.453120] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3457.464351] LustreError: 114972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3458.592600] Lustre: server umount lustre-MDT0001 complete [ 3474.626468] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 3478.827730] Key type lgssc unregistered [ 3479.307631] LNet: 118651:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3479.315047] LNetError: 118651:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3479.332990] LNet: Removed LNI 192.168.203.118@tcp [ 3480.303565] Key type .llcrypt unregistered [ 3480.306488] Key type ._llcrypt unregistered [ 3511.283703] Key type ._llcrypt registered [ 3511.287542] Key type .llcrypt registered [ 3511.389120] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 3512.914666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3512.940289] alg: No test for adler32 (adler32-zlib) [ 3514.020370] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 3514.295519] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 3515.991192] Key type lgssc registered [ 3517.185883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3525.569928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3525.600111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3527.163831] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3527.204696] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.18/24@tcp [ 3527.236626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3531.002155] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3537.379644] LustreError: 120194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3537.417399] LustreError: 120194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3539.161253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3539.430557] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3539.464588] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.18/24@tcp [ 3539.535741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3544.273081] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3549.609503] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3553.177983] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3559.166553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3559.420492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3564.370529] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3569.658864] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3571.702929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3587.064672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3587.069409] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3587.079191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3587.087318] Lustre: Skipped 1 previous similar message [ 3591.639735] Lustre: server umount lustre-OST0000 complete [ 3595.745562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3595.758072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3595.787875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3595.790915] Lustre: Skipped 1 previous similar message [ 3597.286376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3597.293295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3597.306287] Lustre: Skipped 1 previous similar message [ 3601.874299] Lustre: server umount lustre-MDT0000 complete [ 3602.412202] LustreError: 121170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3602.458694] LustreError: 121170:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3606.055390] LustreError: 120174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334163 with bad export cookie 12663137517765466558 [ 3606.065266] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3606.080800] LustreError: 120174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3606.581367] Lustre: server umount lustre-MDT0001 complete [ 3621.084827] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 3624.635268] Key type lgssc unregistered [ 3625.096609] LNet: 123427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3625.113494] LNetError: 123427:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3625.125263] LNet: Removed LNI 192.168.203.118@tcp [ 3625.957853] Key type .llcrypt unregistered [ 3625.961487] Key type ._llcrypt unregistered [ 3652.276982] Key type ._llcrypt registered [ 3652.285813] Key type .llcrypt registered [ 3652.427336] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 3668.145761] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 3669.241965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3669.282962] alg: No test for adler32 (adler32-zlib) [ 3670.588878] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 3670.808164] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 3672.528607] Key type lgssc registered [ 3673.769785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3710.772789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3710.826629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3712.088462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3712.115574] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3712.190435] Lustre: lustre-MDT0000: new disk, initializing [ 3712.257829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3712.280576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3716.780247] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3727.973969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3728.122595] Lustre: 126795:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3728.206562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3728.212289] Lustre: Skipped 1 previous similar message [ 3728.311727] Lustre: lustre-MDT0001: new disk, initializing [ 3728.400094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3728.438919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3728.448990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3733.268515] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3741.090748] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3745.820812] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3753.069782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3753.435184] Lustre: lustre-OST0000: new disk, initializing [ 3753.443323] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3753.455054] Lustre: 128537:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3753.572581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3760.131244] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3762.733731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3762.743665] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3762.855061] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3767.089685] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3771.811742] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3772.165805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3776.766353] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3777.094131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3783.650675] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3783.653658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3783.684562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3783.693724] Lustre: Skipped 1 previous similar message [ 3788.778929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3788.783666] Lustre: Skipped 1 previous similar message [ 3793.376592] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3793.628663] Lustre: server umount lustre-OST0000 complete [ 3799.009039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3799.013129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3799.023923] Lustre: Skipped 2 previous similar messages [ 3799.035329] Lustre: Skipped 1 previous similar message [ 3803.856784] Lustre: server umount lustre-MDT0000 complete [ 3807.832577] LustreError: 126787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334365 with bad export cookie 9209146356485520685 [ 3807.836853] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3807.839946] LustreError: 126787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3808.282079] Lustre: server umount lustre-MDT0001 complete [ 3816.416692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3816.784705] LustreError: 130611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3816.808574] LustreError: 130611:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3816.877133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3821.253742] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3822.049271] LustreError: 130612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3827.170510] LustreError: 130611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3831.935405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3832.388587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3836.297910] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3842.041399] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3845.967388] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3853.140807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3853.547101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3859.989451] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3865.800255] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3870.621302] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3870.861315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3875.467402] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3875.664632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3884.785672] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3884.966246] Lustre: lustre-OST0001: new disk, initializing [ 3884.969633] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3884.975354] Lustre: 133605:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3885.046425] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3891.190815] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3892.281184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3892.310418] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3892.501832] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3898.912230] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3899.133764] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3902.733488] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3902.938660] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3907.227270] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f90487ff800.ost_server_uuid 50 [ 3908.842450] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f90487ff800.ost_server_uuid in IDLE state after 0 sec [ 3918.308918] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3918.337397] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3918.338233] LustreError: Skipped 2 previous similar messages [ 3918.339472] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3918.373415] Lustre: Skipped 1 previous similar message [ 3919.080423] Lustre: server umount lustre-OST0001 complete [ 3923.425919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3923.429195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3923.432325] LustreError: 132206:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3923.435563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3923.435578] Lustre: Skipped 1 previous similar message [ 3923.495302] Lustre: Skipped 1 previous similar message [ 3928.554272] LustreError: 134018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3928.579880] LustreError: 134018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3929.326549] Lustre: server umount lustre-OST0000 complete [ 3933.666971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3933.672631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3933.679031] Lustre: Skipped 1 previous similar message [ 3933.690680] Lustre: Skipped 3 previous similar messages [ 3939.003829] Lustre: server umount lustre-MDT0000 complete [ 3942.368792] LustreError: 130591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334499 with bad export cookie 9209146356485521567 [ 3942.373306] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3942.385657] LustreError: 130591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3942.631759] Lustre: server umount lustre-MDT0001 complete [ 3963.577296] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 3966.592209] Key type lgssc unregistered [ 3966.879633] LNet: 136442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3966.889785] LNetError: 136442:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3967.909791] LNet: Removed LNI 192.168.203.118@tcp [ 3968.960623] Key type .llcrypt unregistered [ 3968.969392] Key type ._llcrypt unregistered [ 4006.959821] Key type ._llcrypt registered [ 4006.963861] Key type .llcrypt registered [ 4007.098329] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 4021.717428] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 4022.901722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4022.925106] alg: No test for adler32 (adler32-zlib) [ 4023.946782] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 4024.116460] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 4025.871859] Key type lgssc registered [ 4026.861926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4059.348813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4059.411280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4060.665846] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4060.719899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4060.861676] Lustre: lustre-MDT0000: new disk, initializing [ 4060.985863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4061.063407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4065.902847] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4075.503757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4075.618265] Lustre: 140566:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4075.662380] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4075.666593] Lustre: Skipped 1 previous similar message [ 4075.733219] Lustre: lustre-MDT0001: new disk, initializing [ 4075.822183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4075.851278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4075.863266] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4080.776763] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4087.971822] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4090.902271] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4096.760827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4097.040926] Lustre: lustre-OST0000: new disk, initializing [ 4097.050225] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4097.063345] Lustre: 142306:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4097.190605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4102.136496] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4105.262135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4105.272422] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4105.372306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4108.092372] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4112.566484] Lustre: Failing over lustre-OST0000 [ 4112.696953] Lustre: server umount lustre-OST0000 complete [ 4129.584470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4129.883237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4129.920041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4129.939334] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4130.911330] Lustre: 138787:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780334672/real 1780334672] req@000000004fae7169 x1866816113637760/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1780334688 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4131.310285] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4131.999905] Lustre: 138789:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780334673/real 1780334673] req@000000004e63c21d x1866816113637888/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1780334689 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4132.026589] Lustre: 138789:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4135.432593] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4135.432903] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 4136.258635] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4136.485816] Lustre: 138787:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780334678/real 1780334678] req@00000000ebd8d0e1 x1866816113639808/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1780334694 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4136.530875] Lustre: 138787:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4140.340797] Lustre: Failing over lustre-MDT0000 [ 4140.527406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4140.541899] Lustre: Skipped 2 previous similar messages [ 4140.713806] Lustre: server umount lustre-MDT0000 complete [ 4142.367301] Lustre: 138788:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780334683/real 1780334683] req@00000000563caac5 x1866816113640832/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1780334699 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4142.426925] Lustre: 138788:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4145.632968] LustreError: 141693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4145.648014] LustreError: 141693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4146.736051] LustreError: 140571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4150.759908] LustreError: 140576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4150.780143] LustreError: 140576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4155.878293] LustreError: 140577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4155.904322] LustreError: 140577:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4160.192036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4160.375289] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4160.586899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4160.631894] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4162.023315] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4164.464228] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4165.635214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4165.644736] Lustre: Skipped 1 previous similar message [ 4165.675535] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4165.761643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4175.841070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4175.847224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4175.852484] Lustre: Skipped 2 previous similar messages [ 4175.866173] Lustre: Skipped 1 previous similar message [ 4180.969629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4180.975606] Lustre: Skipped 1 previous similar message [ 4186.081973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4186.084097] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 4186.088702] Lustre: Skipped 1 previous similar message [ 4186.194840] Lustre: server umount lustre-OST0000 complete [ 4191.207869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4191.209565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4191.225717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4191.264215] Lustre: Skipped 1 previous similar message [ 4195.563985] Lustre: server umount lustre-MDT0000 complete [ 4196.323882] LustreError: 140576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4196.358504] LustreError: 140576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4199.394496] LustreError: 140558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780334756 with bad export cookie 3328275545376785295 [ 4199.400204] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4199.412512] LustreError: 140558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4199.749106] Lustre: server umount lustre-MDT0001 complete [ 4211.540812] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 4214.943419] Key type lgssc unregistered [ 4215.331132] LNet: 145796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4215.346017] LNetError: 145796:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4215.399905] LNet: Removed LNI 192.168.203.118@tcp [ 4216.533832] Key type .llcrypt unregistered [ 4216.535952] Key type ._llcrypt unregistered [ 4253.855181] Key type ._llcrypt registered [ 4253.858210] Key type .llcrypt registered [ 4253.997301] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 4255.819605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4255.874391] alg: No test for adler32 (adler32-zlib) [ 4257.055767] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 4257.373086] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 4259.239217] Key type lgssc registered [ 4260.917578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4268.216299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4268.232307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4269.615640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4273.333676] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4279.778493] LustreError: 147300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4279.792257] LustreError: 147300:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4279.894577] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4283.589075] LustreError: 147329:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4283.599526] LustreError: 147329:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4283.604935] LustreError: 147329:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 4283.939231] Lustre: server umount lustre-MDT0000 complete [ 4295.641929] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 4303.471689] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 4331.063417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4331.287885] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4331.310695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4331.369493] Lustre: lustre-MDT0000: new disk, initializing [ 4331.432931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4331.446136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4335.110755] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4345.241048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4345.396979] Lustre: 150482:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4345.481313] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4345.484881] Lustre: Skipped 1 previous similar message [ 4345.599534] Lustre: lustre-MDT0001: new disk, initializing [ 4345.718772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4345.751719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4345.759438] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4349.767806] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4356.499073] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4359.997491] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4365.187510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4365.443516] Lustre: lustre-OST0000: new disk, initializing [ 4365.446589] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4365.451895] Lustre: 152224:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4365.535780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4370.031791] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4372.009103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4372.021108] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4372.095153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4376.086686] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4380.386551] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4381.923476] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 13:28:57 (1780334937) [ 4596.704737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4596.716427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4596.738060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4597.216860] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4597.227459] Lustre: Skipped 1 previous similar message [ 4602.130113] Lustre: server umount lustre-MDT0000 complete [ 4602.335450] LustreError: 150496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4602.352308] LustreError: 150496:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4605.535397] LustreError: 152223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335163 with bad export cookie 11961255511702998423 [ 4605.535878] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4605.552045] LustreError: 152223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4607.455653] LustreError: 150491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4607.459313] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4607.498397] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4607.507833] Lustre: Skipped 3 previous similar messages [ 4610.463816] LustreError: 150490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4610.464227] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4612.001815] Lustre: server umount lustre-MDT0001 complete [ 4622.290022] Lustre: server umount lustre-OST0000 complete [ 4631.463941] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 4641.833601] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 4677.130751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4677.398924] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4677.460976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4677.635585] Lustre: lustre-MDT0000: new disk, initializing [ 4677.757562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4677.780284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4682.892341] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4694.101266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4694.287689] Lustre: 156730:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4694.412927] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4694.419799] Lustre: Skipped 1 previous similar message [ 4694.551878] Lustre: lustre-MDT0001: new disk, initializing [ 4694.689987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4694.762889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4694.786147] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4700.067991] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4707.480340] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4710.605792] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4716.060823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4716.296677] Lustre: lustre-OST0000: new disk, initializing [ 4716.300413] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4716.309464] Lustre: 158485:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4716.393058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4721.491643] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4721.728931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4721.743318] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4721.836733] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4727.726655] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4732.067820] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4732.275235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4736.398980] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4736.582892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4742.641958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4742.643938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4742.668655] LustreError: Skipped 1 previous similar message [ 4742.717835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4744.669285] Lustre: server umount lustre-OST0000 complete [ 4751.328737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4751.334114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4751.345557] Lustre: Skipped 1 previous similar message [ 4751.360140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4751.371264] Lustre: Skipped 1 previous similar message [ 4754.899493] LustreError: 159877:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000cee43e40) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4755.211176] Lustre: server umount lustre-MDT0000 complete [ 4757.985987] LustreError: 156743:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4758.028731] LustreError: 156743:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4759.877529] LustreError: 157520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335317 with bad export cookie 11961255511703157246 [ 4759.882932] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4759.898425] LustreError: 157520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4760.233484] Lustre: server umount lustre-MDT0001 complete [ 4768.300861] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:35:23 (1780335323) [ 4782.217852] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 4790.320491] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 4822.186060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4822.393172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4822.417961] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4822.492117] Lustre: lustre-MDT0000: new disk, initializing [ 4822.577637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4822.599426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4826.888096] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4838.201256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4838.434817] Lustre: 163057:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4838.615872] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4838.633797] Lustre: Skipped 2 previous similar messages [ 4838.894177] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4843.500786] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4852.409157] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4856.271246] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4863.760143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4864.087864] Lustre: lustre-OST0000: new disk, initializing [ 4864.091827] Lustre: Skipped 1 previous similar message [ 4864.098853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4864.103752] Lustre: Skipped 2 previous similar messages [ 4864.109499] Lustre: 164806:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4871.208291] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4872.281261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4872.295127] Lustre: Skipped 1 previous similar message [ 4872.310161] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4872.459268] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4879.463923] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4887.905969] Lustre: server umount lustre-OST0000 complete [ 4893.155685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4893.159556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4893.172643] Lustre: Skipped 2 previous similar messages [ 4901.432172] Lustre: server umount lustre-MDT0000 complete [ 4903.392941] LustreError: 164261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4903.423251] LustreError: 164261:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4904.415319] Lustre: 146613:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780335445/real 1780335445] req@00000000c46424d0 x1866816358679552/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1780335461 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4904.461379] Lustre: 146613:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4905.278255] LustreError: 163048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335462 with bad export cookie 11961255511703158135 [ 4905.284816] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4905.288275] LustreError: 163048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4905.573881] Lustre: server umount lustre-MDT0001 complete [ 4912.850590] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:37:48 (1780335468) [ 4928.827842] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 4938.551389] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 4975.299925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4975.733448] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4975.788963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4975.972992] Lustre: lustre-MDT0000: new disk, initializing [ 4976.127160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4976.132942] Lustre: Skipped 2 previous similar messages [ 4976.150976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4981.098806] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4992.238588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4992.465739] Lustre: 168981:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4992.672170] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4996.928523] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5005.487478] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5008.888429] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5015.473819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5015.711514] Lustre: 170732:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5021.984302] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5022.389328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5028.612897] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5038.050275] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5038.057654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5038.061580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5038.077247] Lustre: Skipped 2 previous similar messages [ 5038.083407] Lustre: Skipped 3 previous similar messages [ 5041.894517] Lustre: server umount lustre-OST0000 complete [ 5051.583537] Lustre: server umount lustre-MDT0000 complete [ 5053.411405] LustreError: 168993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5053.436651] LustreError: 168993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5055.709729] LustreError: 171436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335613 with bad export cookie 11961255511703159213 [ 5055.715504] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5055.725597] LustreError: 171436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5056.010726] Lustre: server umount lustre-MDT0001 complete [ 5066.762430] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 5070.006567] Key type lgssc unregistered [ 5070.332167] LNet: 172459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5070.343561] LNetError: 172459:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5070.366378] LNet: Removed LNI 192.168.203.118@tcp [ 5071.123977] Key type .llcrypt unregistered [ 5071.125950] Key type ._llcrypt unregistered [ 5095.450644] Key type ._llcrypt registered [ 5095.453339] Key type .llcrypt registered [ 5095.564370] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 5096.858219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5096.999323] alg: No test for adler32 (adler32-zlib) [ 5097.994301] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 5098.293681] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 5100.057970] Key type lgssc registered [ 5101.271277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5108.352115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5108.365623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5109.854934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5114.197982] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5119.969375] LustreError: 173860:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5119.991828] LustreError: 173860:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5122.160389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5122.417779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5126.778218] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5131.805185] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5135.549881] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5141.533218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5141.879872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5147.415714] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5152.515786] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5155.227601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5175.793531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5175.795319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5175.814664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5175.818698] Lustre: Skipped 1 previous similar message [ 5175.827753] Lustre: Skipped 1 previous similar message [ 5180.897602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5180.903320] Lustre: Skipped 1 previous similar message [ 5181.684049] Lustre: server umount lustre-OST0000 complete [ 5186.020098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5186.025241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5186.037398] Lustre: Skipped 1 previous similar message [ 5186.047044] Lustre: Skipped 1 previous similar message [ 5191.136468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5191.150439] Lustre: Skipped 1 previous similar message [ 5191.919285] Lustre: server umount lustre-MDT0000 complete [ 5196.203078] LustreError: 173842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335753 with bad export cookie 17297409738476679000 [ 5196.213678] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5196.215956] LustreError: 173842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5196.548823] Lustre: server umount lustre-MDT0001 complete [ 5207.005398] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 5209.943327] Key type lgssc unregistered [ 5210.307865] LNet: 177061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5210.328048] LNetError: 177061:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5210.347099] LNet: Removed LNI 192.168.203.118@tcp [ 5211.167333] Key type .llcrypt unregistered [ 5211.170243] Key type ._llcrypt unregistered [ 5235.520221] Key type ._llcrypt registered [ 5235.523842] Key type .llcrypt registered [ 5235.657305] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 5237.035565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5237.068223] alg: No test for adler32 (adler32-zlib) [ 5238.187581] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 5238.497311] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 5240.159786] Key type lgssc registered [ 5241.183496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5248.846716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5248.879584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5250.506317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5255.272243] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5260.775296] LustreError: 178462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5260.801205] LustreError: 178462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5264.173220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5264.518977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5268.499896] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5273.873739] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5277.314915] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5283.703113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5284.172682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5289.164753] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5294.551038] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5295.470612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5302.860519] Lustre: lustre-OST0000: Not available for connect from 192.168.203.18@tcp (stopping) [ 5305.833285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5305.834688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5305.873841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5305.877138] LustreError: Skipped 1 previous similar message [ 5305.900418] Lustre: Skipped 1 previous similar message [ 5307.955100] Lustre: lustre-OST0000: Not available for connect from 192.168.203.18@tcp (stopping) [ 5310.973569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5310.982933] Lustre: Skipped 1 previous similar message [ 5316.065907] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5316.076416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5316.094896] Lustre: Skipped 2 previous similar messages [ 5316.257763] Lustre: server umount lustre-OST0000 complete [ 5322.514598] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9f90487f9000.ost_server_uuid 50 [ 5324.429888] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5341.155393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5341.171606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5341.208401] Lustre: Skipped 1 previous similar message [ 5341.220358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5345.775236] LustreError: 178462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5346.444440] Lustre: server umount lustre-MDT0000 complete [ 5350.454943] LustreError: 178442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335908 with bad export cookie 18088424631677573152 [ 5350.458114] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5350.467617] LustreError: 178442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5350.763665] Lustre: server umount lustre-MDT0001 complete [ 5359.045289] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 13:45:14 (1780335914) [ 5364.852033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5365.202222] LustreError: 181941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5365.220856] LustreError: 181941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5365.260102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5369.111689] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5370.340217] LustreError: 181942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5375.461804] LustreError: 181941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5376.599764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5377.034108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5381.080435] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5386.235707] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5389.753670] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5395.393907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5395.674872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5401.440583] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5407.077651] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5411.077148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5412.839452] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5413.063620] Lustre: lustre-OST0001: new disk, initializing [ 5413.067328] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5413.072898] Lustre: 184442:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5413.170375] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5418.394360] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5418.587608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5418.599774] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5418.714260] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5425.278735] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5429.234551] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5429.245993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5429.250764] Lustre: Skipped 2 previous similar messages [ 5429.282485] Lustre: Skipped 2 previous similar messages [ 5434.546256] Lustre: server umount lustre-OST0000 complete [ 5435.947640] LustreError: 183535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5441.100908] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5441.335107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5445.838220] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5446.090969] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5454.820498] LustreError: 183535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5454.840906] LustreError: 183535:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5465.056497] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5465.064766] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5465.067268] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5465.079170] Lustre: Skipped 1 previous similar message [ 5465.110467] Lustre: Skipped 4 previous similar messages [ 5466.512405] Lustre: server umount lustre-OST0001 complete [ 5470.185956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5475.861442] Lustre: server umount lustre-MDT0000 complete [ 5478.820041] LustreError: 181923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336036 with bad export cookie 18088424631677574223 [ 5478.827756] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5478.831583] LustreError: 181923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5479.141631] Lustre: server umount lustre-MDT0001 complete [ 5498.698700] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 13:47:34 (1780336054) [ 5504.315116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5504.617990] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5504.638860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5504.813853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5508.422974] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5515.780895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5515.930427] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5515.991487] Lustre: 187729:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5520.711609] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5525.845772] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5528.979720] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5534.478410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5534.650141] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5540.433129] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5543.021569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5545.567956] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5552.841454] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5552.967503] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5558.921345] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5563.076930] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5564.048762] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5568.482476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5568.499511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5568.509811] Lustre: Skipped 2 previous similar messages [ 5568.520476] Lustre: Skipped 4 previous similar messages [ 5572.511903] Lustre: server umount lustre-OST0000 complete [ 5573.621352] LustreError: 189352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5573.655329] LustreError: 189352:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 5588.449340] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5588.464070] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5594.396417] Lustre: server umount lustre-OST0001 complete [ 5598.178486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5598.184138] LustreError: Skipped 1 previous similar message [ 5603.839635] Lustre: server umount lustre-MDT0000 complete [ 5606.975624] LustreError: 187722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336164 with bad export cookie 18088424631677575630 [ 5606.979990] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5606.984232] LustreError: 187722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5607.332440] Lustre: server umount lustre-MDT0001 complete [ 5628.983386] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 13:49:44 (1780336184) [ 5643.000542] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 5651.293182] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 5680.483394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5680.663529] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5680.689556] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5680.759660] Lustre: lustre-MDT0000: new disk, initializing [ 5680.836209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5680.847040] Lustre: Skipped 3 previous similar messages [ 5680.858222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5684.678744] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5693.764524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5693.866906] Lustre: 195333:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5693.904293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5693.908674] Lustre: Skipped 1 previous similar message [ 5693.996674] Lustre: lustre-MDT0001: new disk, initializing [ 5694.129303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5694.140643] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5697.396493] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5703.363372] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5706.727244] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5712.663180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5712.878840] Lustre: lustre-OST0000: new disk, initializing [ 5712.887151] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5712.898537] Lustre: 197080:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5718.596351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5718.604962] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5718.705756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5718.745842] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5725.088324] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5729.147472] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5729.391652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5733.833965] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5734.028148] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5739.489641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5739.490177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5739.496982] LustreError: Skipped 1 previous similar message [ 5739.504515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5739.506344] Lustre: Skipped 4 previous similar messages [ 5739.527231] Lustre: Skipped 10 previous similar messages [ 5750.239261] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5750.505651] Lustre: server umount lustre-OST0000 complete [ 5759.969392] LustreError: 195346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5759.984328] LustreError: 195346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 5760.008702] Lustre: server umount lustre-MDT0000 complete [ 5763.235272] LustreError: 195328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336320 with bad export cookie 18088424631677577415 [ 5763.246519] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5763.251952] LustreError: 195328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5763.604630] Lustre: server umount lustre-MDT0001 complete [ 5771.182530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5775.245846] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5782.682665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5787.417455] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5792.256394] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5795.574157] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5802.053581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5807.239691] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5812.533762] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5816.389467] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5817.697947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5821.847582] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5822.114963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5827.043573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5827.048278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5827.061198] Lustre: Skipped 3 previous similar messages [ 5838.305116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5838.471443] Lustre: server umount lustre-OST0000 complete [ 5845.049524] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5850.782179] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5881.062096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5881.603242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5881.610205] Lustre: Skipped 5 previous similar messages [ 5890.152222] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5891.822165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.847134] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5891.861244] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.688921] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5898.468302] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5907.427647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5909.255533] Lustre: server umount lustre-OST0000 complete [ 5920.248369] Lustre: server umount lustre-MDT0000 complete [ 5922.785154] LustreError: 199145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5922.796139] LustreError: 199145:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5923.868044] LustreError: 203532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336481 with bad export cookie 18088424631677578297 [ 5923.873681] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5923.880358] LustreError: 203532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5924.250379] Lustre: server umount lustre-MDT0001 complete [ 5932.739711] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 13:54:47 (1780336487) [ 5940.498994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5945.700361] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5954.143535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5959.419354] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5965.540853] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5969.695584] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5978.105661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5985.914168] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 5993.392871] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5998.469779] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5998.829259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6006.006904] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6006.278055] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6014.194663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6014.501785] Lustre: lustre-OST0001: new disk, initializing [ 6014.511412] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6014.519785] Lustre: 207601:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6020.901754] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6023.836550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6023.847653] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6023.957744] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6029.603538] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6034.646364] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6034.944978] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6039.308737] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6039.483768] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6044.649530] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6044.651267] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6044.666337] LustreError: Skipped 2 previous similar messages [ 6044.670130] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6044.701973] Lustre: Skipped 5 previous similar messages [ 6044.704968] Lustre: Skipped 17 previous similar messages [ 6047.580536] Lustre: server umount lustre-OST0001 complete [ 6053.688886] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6053.931068] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6058.478591] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6058.714661] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6088.237492] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6093.803758] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6093.840733] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6093.851634] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6093.878708] Lustre: Skipped 1 previous similar message [ 6095.354404] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6100.740697] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6102.393266] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6118.368424] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6118.583946] Lustre: server umount lustre-OST0001 complete [ 6140.736897] Lustre: server umount lustre-OST0000 complete [ 6152.469411] Lustre: server umount lustre-MDT0000 complete [ 6156.638240] LustreError: 204584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336714 with bad export cookie 18088424631677579263 [ 6156.646613] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6156.664604] LustreError: 204584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6163.129315] Lustre: server umount lustre-MDT0001 complete [ 6187.248272] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 13:59:02 (1780336742) [ 6194.460902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6194.837199] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6194.886779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6195.130536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6195.138874] Lustre: Skipped 5 previous similar messages [ 6199.837277] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6210.053924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6210.182773] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6210.231579] Lustre: 212552:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6215.109697] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6220.992507] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6225.147473] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6234.950037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6235.289716] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6243.550910] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6251.115222] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6261.385149] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6261.560236] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6268.936508] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6275.414274] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6277.486778] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6282.624671] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6282.897254] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6287.104126] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6287.341230] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6291.058220] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f904675e800.ost_server_uuid 50 [ 6292.738275] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f904675e800.ost_server_uuid in IDLE state after 0 sec [ 6293.741714] Lustre: Permanently deactivating lustre-OST0001 [ 6293.755695] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6297.732683] Lustre: Permanently reactivating lustre-OST0001 [ 6297.763277] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6297.770404] Lustre: Skipped 2 previous similar messages [ 6303.714381] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6303.722086] LustreError: Skipped 2 previous similar messages [ 6303.727513] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6303.741312] Lustre: Skipped 6 previous similar messages [ 6304.249178] LustreError: 214170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6304.285915] LustreError: 214170:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 39 previous similar messages [ 6304.412188] Lustre: server umount lustre-OST0001 complete [ 6314.422215] Lustre: server umount lustre-OST0000 complete [ 6324.714847] Lustre: server umount lustre-MDT0000 complete [ 6328.755749] LustreError: 212544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336886 with bad export cookie 18088424631677580551 [ 6328.768825] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6328.772319] LustreError: 212544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6329.252052] Lustre: server umount lustre-MDT0001 complete [ 6353.528221] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 14:01:48 (1780336908) [ 6361.897961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6362.279762] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6362.312163] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6362.322924] Lustre: Skipped 2 previous similar messages [ 6367.086940] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6376.400437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6376.524122] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6376.553912] Lustre: 218674:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6381.788210] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6387.673966] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6391.711133] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6398.888128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6399.114623] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6399.149452] Lustre: Permanently deactivating lustre-OST0000 [ 6399.177221] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6406.677342] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6406.952854] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6412.519625] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6419.618546] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6419.678382] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6425.943507] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6428.833800] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6431.413779] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6433.775729] Lustre: Permanently reactivating lustre-OST0000 [ 6433.784752] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6433.788369] Lustre: Skipped 2 previous similar messages [ 6435.607758] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6435.680843] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6435.718811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6435.744414] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6435.766598] Lustre: Skipped 1 previous similar message [ 6435.769963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6454.396791] Lustre: server umount lustre-OST0001 complete [ 6464.438443] Lustre: server umount lustre-OST0000 complete [ 6474.761847] Lustre: server umount lustre-MDT0000 complete [ 6478.965622] LustreError: 218665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337036 with bad export cookie 18088424631677582224 [ 6478.971066] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6478.984218] LustreError: 218665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6479.482556] Lustre: server umount lustre-MDT0001 complete [ 6492.189845] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 6495.401477] Key type lgssc unregistered [ 6495.832604] LNet: 223087:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6495.846220] LNetError: 223087:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6495.879312] LNet: Removed LNI 192.168.203.118@tcp [ 6496.849957] Key type .llcrypt unregistered [ 6496.854955] Key type ._llcrypt unregistered [ 6520.406055] Key type ._llcrypt registered [ 6520.408790] Key type .llcrypt registered [ 6520.602557] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 6521.864226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6521.900975] alg: No test for adler32 (adler32-zlib) [ 6522.886476] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 6523.084390] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 6524.807155] Key type lgssc registered [ 6526.120808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6537.409189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6537.465499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6538.965747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6543.515583] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6548.450765] LustreError: 224537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6548.476505] LustreError: 224537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6552.433031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6552.540298] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6552.571780] Lustre: 224524:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6552.585446] Lustre: Permanently deactivating lustre-MDT0001 [ 6552.602157] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6552.887703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6555.599034] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6555.614230] LustreError: 224568:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6555.635162] LustreError: 224568:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6555.648235] LustreError: 224568:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 6557.306827] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6562.487203] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6566.586496] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6573.010034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6573.448230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6577.584307] LustreError: 226153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6577.610830] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6579.482762] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6582.770099] LustreError: 226394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6583.820676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6585.595367] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6588.897207] LustreError: 226153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6588.923053] LustreError: 226153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6593.427327] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6593.698114] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6599.171491] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6599.182208] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6599.735479] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6606.879865] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6610.949353] Lustre: Permanently reactivating lustre-MDT0001 [ 6610.986436] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6610.999829] Lustre: Skipped 2 previous similar messages [ 6615.639571] Lustre: lustre-MDT0001: Client 7c3edef7-edfd-4d7d-82bc-7d6efd4875af (at 192.168.203.18@tcp) reconnecting [ 6619.665774] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6621.451277] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6627.533238] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6627.755880] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6632.355623] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6632.594780] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6635.109622] Lustre: Permanently deactivating lustre-MDT0001 [ 6635.151501] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6635.157759] Lustre: Skipped 2 previous similar messages [ 6641.281445] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6650.343855] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6650.344601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6650.352483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6650.388751] Lustre: Skipped 3 previous similar messages [ 6651.400492] Lustre: server umount lustre-MDT0000 complete [ 6655.075622] LustreError: 226172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337212 with bad export cookie 14877792203310230006 [ 6655.079980] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6655.098752] LustreError: 226172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6655.463799] Lustre: server umount lustre-MDT0001 complete [ 6661.578317] Lustre: server umount lustre-OST0000 complete [ 6675.061215] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 14:07:09 (1780337229) [ 6688.522575] Lustre: server umount lustre-OST0001 complete [ 6696.747945] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 6711.113053] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 6752.152623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6752.477553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6752.504121] Lustre: Skipped 2 previous similar messages [ 6752.540323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6752.691025] Lustre: lustre-MDT0000: new disk, initializing [ 6752.837418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6752.880715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6757.459726] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6770.083261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6770.279528] Lustre: 232194:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6770.374888] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6770.384061] Lustre: Skipped 1 previous similar message [ 6770.523961] Lustre: lustre-MDT0001: new disk, initializing [ 6770.650171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6770.711790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6770.735207] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6775.759881] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6785.127590] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6789.533537] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6796.875299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6797.159472] Lustre: lustre-OST0000: new disk, initializing [ 6797.169099] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6797.188814] Lustre: 233937:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6797.306060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6803.415708] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6804.591494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6804.610053] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6804.726427] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6812.824766] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6816.063437] LustreError: 232200:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6826.104037] LustreError: 232200:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6826.140395] LustreError: 233214:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6826.900704] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6827.076425] Lustre: lustre-OST0001: new disk, initializing [ 6827.082886] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6827.090710] Lustre: 234939:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6833.836813] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6836.191150] LustreError: 233214:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6837.348729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6837.358369] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6837.454872] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6843.331178] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6845.321375] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6847.985648] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6847.986288] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6847.995637] LustreError: Skipped 1 previous similar message [ 6848.015034] Lustre: Skipped 1 previous similar message [ 6848.017690] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6848.017698] Lustre: Skipped 3 previous similar messages [ 6853.089221] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6853.097116] Lustre: Skipped 2 previous similar messages [ 6853.700425] Lustre: server umount lustre-OST0001 complete [ 6858.214343] LustreError: 234348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6858.232377] LustreError: 234348:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6868.453290] LustreError: 233921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6868.481061] LustreError: 233921:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6871.520365] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6871.524675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6871.529862] LustreError: Skipped 1 previous similar message [ 6871.531072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6871.579658] Lustre: Skipped 1 previous similar message [ 6878.689619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6878.706268] Lustre: Skipped 4 previous similar messages [ 6885.343182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6885.660566] Lustre: server umount lustre-OST0000 complete [ 6893.543804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6893.561996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6893.598165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6893.611890] Lustre: Skipped 2 previous similar messages [ 6896.356781] Lustre: server umount lustre-MDT0000 complete [ 6899.169338] LustreError: 232205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6899.188093] LustreError: 232205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 6900.670778] LustreError: 232971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337458 with bad export cookie 14877792203310232820 [ 6900.674196] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6900.687733] LustreError: 232971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6907.336598] Lustre: server umount lustre-MDT0001 complete [ 6918.122622] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 6921.158838] Key type lgssc unregistered [ 6921.443189] LNet: 236845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6921.456488] LNetError: 236845:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6921.473606] LNet: Removed LNI 192.168.203.118@tcp [ 6922.201942] Key type .llcrypt unregistered [ 6922.205511] Key type ._llcrypt unregistered [ 6960.193113] Key type ._llcrypt registered [ 6960.196302] Key type .llcrypt registered [ 6960.341456] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 6961.507642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6961.538731] alg: No test for adler32 (adler32-zlib) [ 6962.623926] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 6962.903990] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 6964.591205] Key type lgssc registered [ 6965.770553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6973.938451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6973.956991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6975.284404] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6975.333989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6975.475598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6979.484303] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6988.593720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6988.753531] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6988.787993] Lustre: 239036:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6989.087587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6993.648547] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6999.181627] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7003.350614] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7010.043743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7010.199519] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7010.408641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7016.396584] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7017.691669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7023.047864] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7079.396743] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7079.426732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7079.427266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7084.106617] Lustre: server umount lustre-OST0000 complete [ 7090.294157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7107.907387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7108.201965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7116.312890] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7119.501228] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7119.530354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7119.532289] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7119.562447] Lustre: Skipped 1 previous similar message [ 7122.324467] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7165.928278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7165.928813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7165.936100] LustreError: Skipped 1 previous similar message [ 7165.937850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7165.949054] Lustre: Skipped 2 previous similar messages [ 7165.975311] Lustre: Skipped 2 previous similar messages [ 7167.486282] Lustre: server umount lustre-OST0000 complete [ 7173.612413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7173.628849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7173.658693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7176.160565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7176.163288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7177.854451] Lustre: server umount lustre-MDT0000 complete [ 7182.310535] LustreError: 239028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337739 with bad export cookie 13324350083368130597 [ 7182.317255] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7182.321508] LustreError: 239028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7182.907714] Lustre: server umount lustre-MDT0001 complete [ 7193.944592] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 7196.797295] Key type lgssc unregistered [ 7197.129774] LNet: 243991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7197.141576] LNetError: 243991:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7197.160344] LNet: Removed LNI 192.168.203.118@tcp [ 7197.997665] Key type .llcrypt unregistered [ 7198.002131] Key type ._llcrypt unregistered [ 7223.916690] Key type ._llcrypt registered [ 7223.921876] Key type .llcrypt registered [ 7224.064992] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 7225.376546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7225.411839] alg: No test for adler32 (adler32-zlib) [ 7226.561848] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 7226.860881] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 7228.584814] Key type lgssc registered [ 7230.402555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7240.665284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7240.724625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7242.238793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7246.888200] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7252.467748] LustreError: 245394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7252.490062] LustreError: 245394:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7255.314699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7255.764609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7260.282946] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7265.882144] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7269.699926] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7276.594846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7276.934246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7282.755091] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7288.252773] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7288.306904] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7316.960236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7316.975512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7316.992221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7319.010478] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7319.020213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7319.030765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7320.467790] Lustre: server umount lustre-OST0000 complete [ 7327.203336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7327.214477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7327.237107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7327.245691] Lustre: Skipped 1 previous similar message [ 7329.249394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7329.249906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7329.282294] Lustre: Skipped 1 previous similar message [ 7331.212396] Lustre: server umount lustre-MDT0000 complete [ 7334.369721] LustreError: 245393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7334.410799] LustreError: 245393:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7336.604865] LustreError: 245375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337894 with bad export cookie 7478869794227877924 [ 7336.609248] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7336.632895] LustreError: 245375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7337.354520] Lustre: server umount lustre-MDT0001 complete [ 7350.381590] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 7353.345365] Key type lgssc unregistered [ 7353.657688] LNet: 249343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7353.666900] LNetError: 249343:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7354.736473] LNet: Removed LNI 192.168.203.118@tcp [ 7355.734445] Key type .llcrypt unregistered [ 7355.738242] Key type ._llcrypt unregistered [ 7373.966514] Key type ._llcrypt registered [ 7373.969203] Key type .llcrypt registered [ 7374.059817] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 7375.215233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7375.368054] alg: No test for adler32 (adler32-zlib) [ 7376.502525] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 7376.793734] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 7378.511223] Key type lgssc registered [ 7379.957579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7388.159665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7388.182944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7389.725693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7394.487206] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7399.906387] LustreError: 250547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7399.916638] LustreError: 250547:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7402.718671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7403.050336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7407.172930] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7412.305878] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7415.675313] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7421.951874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7422.449433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7428.676980] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7434.435116] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7434.819575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7448.038525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7448.052625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7448.081821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7450.096411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7450.098175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7450.098794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7450.105402] Lustre: Skipped 1 previous similar message [ 7451.430722] Lustre: server umount lustre-OST0000 complete [ 7455.206780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7455.220601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7455.225622] Lustre: Skipped 1 previous similar message [ 7460.322037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7460.331083] Lustre: Skipped 1 previous similar message [ 7463.196403] Lustre: server umount lustre-MDT0000 complete [ 7465.440503] LustreError: 250547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7465.472662] LustreError: 250547:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7467.253895] LustreError: 250528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338024 with bad export cookie 9061499674418138678 [ 7467.259660] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7467.265821] LustreError: 250528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7467.881519] Lustre: server umount lustre-MDT0001 complete [ 7479.543629] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 7482.352571] Key type lgssc unregistered [ 7482.748325] LNet: 253868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7482.757568] LNetError: 253868:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7482.777136] LNet: Removed LNI 192.168.203.118@tcp [ 7483.898214] Key type .llcrypt unregistered [ 7483.901283] Key type ._llcrypt unregistered [ 7508.816732] Key type ._llcrypt registered [ 7508.819613] Key type .llcrypt registered [ 7508.937435] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 7510.333283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7510.373648] alg: No test for adler32 (adler32-zlib) [ 7511.636469] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 7511.989067] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 7513.703396] Key type lgssc registered [ 7514.945147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7523.020147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7523.052256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7524.607903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7528.836865] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7535.082572] LustreError: 255268:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7535.109254] LustreError: 255268:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7538.941383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7539.413054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7543.954297] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7549.384786] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7553.665633] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7560.556033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7560.954039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7567.236775] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7570.162377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7573.272281] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7600.096393] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7600.099333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7600.113305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7601.126489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7601.127712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7601.136962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7601.156616] Lustre: Skipped 1 previous similar message [ 7606.251814] LustreError: 256862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7606.268296] LustreError: 256862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7606.386436] Lustre: server umount lustre-OST0000 complete [ 7610.850971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7610.854948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7610.870689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7615.755927] Lustre: server umount lustre-MDT0000 complete [ 7616.484177] LustreError: 255269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7616.515272] LustreError: 255269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7618.821170] LustreError: 255250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338176 with bad export cookie 5157774528202730321 [ 7618.828649] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7618.838461] LustreError: 255250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7619.129604] Lustre: server umount lustre-MDT0001 complete [ 7628.603476] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 7631.459741] Key type lgssc unregistered [ 7631.761892] LNet: 259217:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7631.768744] LNetError: 259217:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7631.790192] LNet: Removed LNI 192.168.203.118@tcp [ 7632.883623] Key type .llcrypt unregistered [ 7632.888521] Key type ._llcrypt unregistered [ 7650.910097] Key type ._llcrypt registered [ 7650.913464] Key type .llcrypt registered [ 7651.048861] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 7652.108325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7652.157211] alg: No test for adler32 (adler32-zlib) [ 7653.341301] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 7653.651852] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 7655.343177] Key type lgssc registered [ 7657.012867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7670.484647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7670.504219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7672.226250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7677.497354] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7681.526526] LustreError: 260438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7681.555517] LustreError: 260438:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7686.626221] LustreError: 260440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7686.854414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7687.322822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7691.795144] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7696.923859] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7702.008997] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7709.529263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7709.885791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7715.200867] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7717.049929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7720.324148] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7732.712645] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7732.717178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7732.722303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7732.757125] Lustre: Skipped 1 previous similar message [ 7734.247831] Lustre: server umount lustre-OST0000 complete [ 7742.954151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7742.965453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7742.976321] Lustre: Skipped 1 previous similar message [ 7742.990112] Lustre: Skipped 2 previous similar messages [ 7745.113700] Lustre: server umount lustre-MDT0000 complete [ 7748.087782] LustreError: 260437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7748.115895] LustreError: 260437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7748.968329] LustreError: 260403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338306 with bad export cookie 4933776069368554647 [ 7748.975260] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7748.977734] LustreError: 260403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7749.420196] Lustre: server umount lustre-MDT0001 complete [ 7764.284622] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 7767.979648] Key type lgssc unregistered [ 7768.319376] LNet: 263783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7768.327135] LNetError: 263783:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7768.371978] LNet: Removed LNI 192.168.203.118@tcp [ 7769.499275] Key type .llcrypt unregistered [ 7769.501828] Key type ._llcrypt unregistered [ 7780.513297] Key type ._llcrypt registered [ 7780.519562] Key type .llcrypt registered [ 7780.655203] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7797.475887] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 7816.239284] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 7817.869213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7818.072255] alg: No test for adler32 (adler32-zlib) [ 7819.332654] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 7819.607703] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 7821.311266] Key type lgssc registered [ 7822.762420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7856.920522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7856.967202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7858.291910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7858.365160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7858.442738] Lustre: lustre-MDT0000: new disk, initializing [ 7858.557123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7858.585674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7862.989786] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7874.387528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7874.523339] Lustre: 267286:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7874.633415] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7874.637340] Lustre: Skipped 1 previous similar message [ 7874.744974] Lustre: lustre-MDT0001: new disk, initializing [ 7874.868125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7874.958536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7874.972575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7879.708628] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7889.356937] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7893.329239] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7900.916472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7901.198971] Lustre: lustre-OST0000: new disk, initializing [ 7901.210327] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7901.224744] Lustre: 269029:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7901.342411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7906.721503] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7910.499075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7910.516694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7910.608277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7914.032218] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7918.438296] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7918.637462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7922.751666] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7922.951299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7926.240930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7926.241878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7926.254200] LustreError: Skipped 1 previous similar message [ 7926.259173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7926.284171] Lustre: Skipped 1 previous similar message [ 7931.177679] Lustre: server umount lustre-OST0000 complete [ 7936.483409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7936.484339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7936.486198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7936.486205] Lustre: Skipped 1 previous similar message [ 7936.499891] Lustre: Skipped 1 previous similar message [ 7940.613982] Lustre: server umount lustre-MDT0000 complete [ 7944.228620] LustreError: 267277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338501 with bad export cookie 8566875205880433078 [ 7944.231643] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7944.242055] LustreError: 267277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7944.629904] Lustre: server umount lustre-MDT0001 complete [ 7952.710691] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 14:28:27 (1780338507) [ 7960.246642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7960.702215] LustreError: 271290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7960.728612] LustreError: 271290:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7960.769935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7965.922813] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7966.182218] LustreError: 271291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7971.304233] LustreError: 271290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7974.038542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7974.316103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7979.238842] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7985.684517] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7990.659862] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8000.785632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8001.476810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8009.302917] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8016.117255] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8029.154441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8029.160927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8029.163242] Lustre: Skipped 1 previous similar message [ 8034.275834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8034.288582] Lustre: Skipped 1 previous similar message [ 8035.450297] Lustre: server umount lustre-OST0000 complete [ 8039.393539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8039.397215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8039.406137] Lustre: Skipped 2 previous similar messages [ 8039.426647] Lustre: Skipped 2 previous similar messages [ 8044.706157] Lustre: server umount lustre-MDT0000 complete [ 8048.046660] LustreError: 271271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338605 with bad export cookie 8566875205880433967 [ 8048.048540] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8048.062148] LustreError: 271271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8048.333495] Lustre: server umount lustre-MDT0001 complete [ 8058.865846] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 8061.300618] Key type lgssc unregistered [ 8061.569895] LNet: 274490:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8061.585052] LNetError: 274490:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8062.633918] LNet: Removed LNI 192.168.203.118@tcp [ 8063.425412] Key type .llcrypt unregistered [ 8063.431418] Key type ._llcrypt unregistered [ 8087.864988] Key type ._llcrypt registered [ 8087.870160] Key type .llcrypt registered [ 8088.017699] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 8102.438464] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 8103.550822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8103.667064] alg: No test for adler32 (adler32-zlib) [ 8104.795451] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 8105.001575] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 8106.703165] Key type lgssc registered [ 8107.961846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8151.356317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8151.382634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8152.815473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8152.893694] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8153.084617] Lustre: lustre-MDT0000: new disk, initializing [ 8153.240198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8153.266131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8157.501341] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8168.190109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8168.330769] Lustre: 278283:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8168.437593] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8168.445047] Lustre: Skipped 1 previous similar message [ 8168.524838] Lustre: lustre-MDT0001: new disk, initializing [ 8168.636478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8168.668764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8168.682537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8174.078468] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8181.222962] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8185.196408] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8191.703103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8191.972441] Lustre: lustre-OST2710: new disk, initializing [ 8191.977337] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8191.985607] Lustre: 280026:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8192.084426] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8199.029547] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8199.812071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8199.818816] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8200.018561] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8207.153316] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8213.466973] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8213.643243] Lustre: lustre-OST03e8: new disk, initializing [ 8213.646879] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8213.652838] Lustre: 280976:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8213.729478] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8219.040419] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8221.243934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8221.253968] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8221.323194] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8224.188118] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8230.547956] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8230.728503] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8235.373524] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8235.606766] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8239.859000] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8240.087411] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8243.958375] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8244.119976] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8250.505192] Lustre: server umount lustre-MDT0000 complete [ 8250.851410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8250.861577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8250.877844] LustreError: 278296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8250.895973] LustreError: 278296:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8252.386937] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8252.389470] LustreError: 278295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8252.403309] Lustre: Skipped 2 previous similar messages [ 8252.444463] LustreError: 278295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8253.783484] LustreError: 279063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338811 with bad export cookie 12153929824112983220 [ 8253.796469] LustreError: 279063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8253.808519] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8254.139046] Lustre: server umount lustre-MDT0001 complete [ 8267.965847] Lustre: server umount lustre-OST2710 complete [ 8281.652277] Lustre: server umount lustre-OST03e8 complete [ 8301.077272] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 8312.588180] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 8347.238728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8347.497262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8347.534911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8347.616932] Lustre: lustre-MDT0000: new disk, initializing [ 8347.680667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8347.695903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8351.850297] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8361.651136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8361.783225] Lustre: 286010:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8361.878641] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8361.889389] Lustre: Skipped 1 previous similar message [ 8362.001510] Lustre: lustre-MDT0001: new disk, initializing [ 8362.150767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8362.202392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8362.219441] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8366.486079] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8373.171670] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8376.500142] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8382.841619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8383.231468] Lustre: lustre-OST0000: new disk, initializing [ 8383.238326] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8383.246720] Lustre: 287752:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8383.323659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8389.441675] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8390.731753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8390.765270] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8390.963424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8397.406608] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8401.745710] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8401.953509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8406.776111] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8407.012984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8411.630651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8411.641971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8411.652219] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8411.660516] Lustre: Skipped 1 previous similar message [ 8411.681891] Lustre: Skipped 1 previous similar message [ 8416.736727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8416.746014] Lustre: Skipped 1 previous similar message [ 8421.882344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8421.892021] Lustre: Skipped 1 previous similar message [ 8423.391145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8423.627992] Lustre: server umount lustre-OST0000 complete [ 8429.030136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8429.039516] LustreError: Skipped 1 previous similar message [ 8429.042651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8429.056223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8433.435403] Lustre: server umount lustre-MDT0000 complete [ 8437.192715] LustreError: 286002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338994 with bad export cookie 12153929824112984907 [ 8437.204516] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8437.212554] LustreError: 286002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8437.574483] Lustre: server umount lustre-MDT0001 complete [ 8445.922630] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 14:36:41 (1780339001) [ 8447.623145] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8449.429354] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 14:36:44 (1780339004) [ 8452.597264] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 8479.628262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8479.913611] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8479.960557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8480.158825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8484.330448] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8492.458355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8492.569787] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8492.621543] Lustre: 291446:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8497.333045] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8518.828491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8519.105287] Lustre: Denying initial registration attempt from nid 192.168.203.118@tcp, specified as failover [ 8519.119662] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8519.126835] LustreError: 292865:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 8519.133253] LustreError: 292865:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 8519.139730] LustreError: 292865:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8519.242652] Lustre: server umount lustre-OST0000 complete [ 8519.247298] LustreError: 292865:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8528.865076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8528.875318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8528.895821] Lustre: Skipped 1 previous similar message [ 8528.902239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8528.917336] Lustre: Skipped 2 previous similar messages [ 8532.509333] Lustre: server umount lustre-MDT0000 complete [ 8534.502503] LustreError: 291458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8534.526227] LustreError: 291458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8536.101341] LustreError: 291438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339093 with bad export cookie 12153929824112985789 [ 8536.110947] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8536.119419] LustreError: 291438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8536.362506] Lustre: server umount lustre-MDT0001 complete [ 8547.746815] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 8551.857461] Key type lgssc unregistered [ 8552.175185] LNet: 293969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8552.195776] LNetError: 293969:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8552.220395] LNet: Removed LNI 192.168.203.118@tcp [ 8553.554139] Key type .llcrypt unregistered [ 8553.557676] Key type ._llcrypt unregistered [ 8566.026303] Key type ._llcrypt registered [ 8566.029401] Key type .llcrypt registered [ 8566.193376] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 8567.160690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8567.301476] alg: No test for adler32 (adler32-zlib) [ 8568.442034] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 8568.639955] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 8570.279236] Key type lgssc registered [ 8571.620529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8617.101858] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 8626.655813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8626.685802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8628.137834] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8628.189156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8628.361423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8633.329936] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8641.814179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8641.902740] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8641.941960] Lustre: 296699:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8642.241987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8646.390917] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8652.133646] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8656.269897] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8676.655351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8676.833727] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8677.012446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8683.190162] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8689.037524] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8693.216377] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8693.222914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8693.238173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8693.731640] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8693.734663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8693.760424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8697.655339] Lustre: server umount lustre-OST0000 complete [ 8703.970501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8703.973237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8703.984744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8703.999384] Lustre: Skipped 1 previous similar message [ 8704.014503] Lustre: Skipped 1 previous similar message [ 8707.458696] Lustre: server umount lustre-MDT0000 complete [ 8711.399439] LustreError: 296692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339268 with bad export cookie 8905490976003238185 [ 8711.410165] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8711.777859] Lustre: server umount lustre-MDT0001 complete [ 8721.983441] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 8724.548547] Key type lgssc unregistered [ 8724.835543] LNet: 299917:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8724.849737] LNetError: 299917:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8724.867660] LNet: Removed LNI 192.168.203.118@tcp [ 8725.602134] Key type .llcrypt unregistered [ 8725.604318] Key type ._llcrypt unregistered [ 8757.485221] Key type ._llcrypt registered [ 8757.487466] Key type .llcrypt registered [ 8757.627699] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 8758.745828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8758.799497] alg: No test for adler32 (adler32-zlib) [ 8759.910919] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 8760.193198] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 8761.927274] Key type lgssc registered [ 8763.387794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8773.214907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8773.273751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8774.977335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8780.017456] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8785.380988] LustreError: 301318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8785.400738] LustreError: 301318:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8789.352025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8789.680587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8794.280038] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8800.597629] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8804.854258] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8814.003470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8814.335341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8820.880650] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8826.616922] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8835.552755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8835.558174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8835.579979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8836.075289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8836.103860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8836.118873] Lustre: Skipped 2 previous similar messages [ 8838.748089] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.18@tcp (stopping) [ 8841.199072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8841.211159] Lustre: Skipped 1 previous similar message [ 8841.947141] Lustre: server umount lustre-MDT0000 complete [ 8843.819848] LustreError: 301313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8846.311342] LustreError: 303470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8846.358644] LustreError: 303470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8846.429313] LustreError: 301297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339403 with bad export cookie 13603137239673074786 [ 8846.461883] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8848.938768] LustreError: 301315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8848.975380] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.18@tcp (stopping) [ 8851.426968] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8851.434596] Lustre: Skipped 1 previous similar message [ 8852.856518] Lustre: server umount lustre-MDT0001 complete [ 8859.360141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8867.653821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8871.907918] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbcc80d3f399a104d [ 8871.923462] Lustre: MGC192.168.203.118@tcp: Connection restored to 0@lo (at 0@lo) [ 8872.090702] LustreError: 302923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8872.109421] LustreError: 302923:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8872.202054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8872.268371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8876.269213] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8881.632992] LustreError: 302924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8881.647536] LustreError: 302924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8882.664724] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8884.391634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8884.855407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8889.431803] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8892.400408] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8894.849193] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8896.337446] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8900.078520] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8901.723633] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8907.750950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8907.760468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8907.788928] Lustre: Skipped 1 previous similar message [ 8907.812454] Lustre: Skipped 2 previous similar messages [ 8912.865668] LustreError: 302912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8912.883452] LustreError: 302912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8913.297614] Lustre: server umount lustre-OST0000 complete [ 8917.989795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8918.011084] Lustre: Skipped 1 previous similar message [ 8928.226846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8928.234232] Lustre: Skipped 5 previous similar messages [ 8932.319259] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8932.510383] Lustre: server umount lustre-MDT0000 complete [ 8936.705385] LustreError: 301297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339494 with bad export cookie 13603137239673081933 [ 8936.731128] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8936.732669] LustreError: 301297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8937.149380] Lustre: server umount lustre-MDT0001 complete [ 8948.369616] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 8950.917927] Key type lgssc unregistered [ 8951.206612] LNet: 307000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8951.215804] LNetError: 307000:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8951.236784] LNet: Removed LNI 192.168.203.118@tcp [ 8952.051108] Key type .llcrypt unregistered [ 8952.053921] Key type ._llcrypt unregistered [ 8980.508210] Key type ._llcrypt registered [ 8980.511626] Key type .llcrypt registered [ 8980.625489] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 8981.449202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8981.558036] alg: No test for adler32 (adler32-zlib) [ 8982.751293] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 8983.030269] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 8984.847155] Key type lgssc registered [ 8986.258272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8994.779316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8994.811903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8996.482623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9001.014304] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9007.076643] LustreError: 308402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9007.108963] LustreError: 308402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9008.951975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9009.411758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9014.152409] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9019.717926] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9023.153611] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9026.251914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9026.254758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9026.276067] Lustre: Skipped 1 previous similar message [ 9026.287461] Lustre: Skipped 1 previous similar message [ 9031.649499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9031.654782] Lustre: Skipped 1 previous similar message [ 9032.539816] Lustre: server umount lustre-MDT0000 complete [ 9033.700453] LustreError: 308398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9033.712264] LustreError: 308398:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9035.748086] LustreError: 308383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339593 with bad export cookie 18328904603212326902 [ 9035.762239] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9035.766233] LustreError: 308383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9036.309932] Lustre: server umount lustre-MDT0001 complete [ 9052.483521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9052.966159] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9053.016551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9053.235431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9058.008362] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9067.479391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9067.570440] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9067.625490] Lustre: 310763:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9067.972531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9073.128524] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9079.025758] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9083.229316] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9093.994578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9094.222420] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9094.240925] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9094.252672] LustreError: 312422:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 9094.262228] LustreError: 312422:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 9094.275645] LustreError: 312422:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9094.365356] Lustre: server umount lustre-OST0000 complete [ 9094.367672] LustreError: 312422:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9105.024216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9105.259890] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9105.466732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9112.277874] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9113.826279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9117.901199] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9126.319292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9126.439267] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9126.524397] Lustre: lustre-OST0001: new disk, initializing [ 9126.527386] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9126.536332] Lustre: 313849:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9126.635406] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9133.144270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9133.150708] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9133.224628] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9133.622890] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9139.365580] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9143.788642] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9143.798277] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9143.807642] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9143.830859] Lustre: Skipped 1 previous similar message [ 9148.859668] Lustre: server umount lustre-OST0001 complete [ 9148.901063] LustreError: 314155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9148.924202] LustreError: 314155:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9154.022130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9154.022811] LustreError: 314155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9154.033050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9154.066115] Lustre: Skipped 1 previous similar message [ 9154.133973] Lustre: Skipped 2 previous similar messages [ 9159.143221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9159.146462] LustreError: 312887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9159.157874] Lustre: Skipped 1 previous similar message [ 9159.214904] LustreError: 312887:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9159.515298] Lustre: server umount lustre-OST0000 complete [ 9164.258175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9169.381162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9169.389296] Lustre: Skipped 3 previous similar messages [ 9170.090791] Lustre: server umount lustre-MDT0000 complete [ 9174.497224] LustreError: 312914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9174.516885] LustreError: 312914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9174.566483] LustreError: 310756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339732 with bad export cookie 18328904603212327448 [ 9174.576410] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9174.581971] LustreError: 310756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9175.213080] Lustre: server umount lustre-MDT0001 complete [ 9187.287835] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 9190.417058] Key type lgssc unregistered [ 9190.732777] LNet: 315678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9190.739293] LNetError: 315678:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9190.764650] LNet: Removed LNI 192.168.203.118@tcp [ 9191.554233] Key type .llcrypt unregistered [ 9191.560996] Key type ._llcrypt unregistered [ 9263.951665] Key type ._llcrypt registered [ 9263.954577] Key type .llcrypt registered [ 9264.061501] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 9278.779829] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 9280.352380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9280.369213] alg: No test for adler32 (adler32-zlib) [ 9281.484912] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 9281.688661] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 9283.399192] Key type lgssc registered [ 9284.555184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9321.480669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9321.529322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9322.809800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9322.859853] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9322.970468] Lustre: lustre-MDT0000: new disk, initializing [ 9323.042200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9323.058806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9327.909722] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9339.011646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9339.091195] Lustre: 320580:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9339.151544] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9339.156204] Lustre: Skipped 1 previous similar message [ 9339.264836] Lustre: lustre-MDT0001: new disk, initializing [ 9339.378368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9339.441751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9339.462487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9344.052129] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9351.705985] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9355.833255] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9364.178610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9364.457751] Lustre: lustre-OST0000: new disk, initializing [ 9364.461852] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9364.473747] Lustre: 322320:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9364.582717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9371.235810] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9373.310339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9373.329332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9373.401666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9379.988792] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9385.245092] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9385.532450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9390.250811] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9390.527574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9394.144839] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9394.157133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9394.169642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9394.179478] Lustre: Skipped 1 previous similar message [ 9394.204583] Lustre: Skipped 1 previous similar message [ 9399.148426] Lustre: server umount lustre-OST0000 complete [ 9404.385142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9404.389591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9404.405958] Lustre: Skipped 1 previous similar message [ 9409.507198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9409.519589] Lustre: Skipped 2 previous similar messages [ 9409.894853] Lustre: server umount lustre-MDT0000 complete [ 9413.488387] LustreError: 320570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339971 with bad export cookie 10357137466345681551 [ 9413.496865] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9413.498204] LustreError: 320570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9413.971223] Lustre: server umount lustre-MDT0001 complete [ 9423.724909] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 14:52:58 (1780339978) [ 9431.528591] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 14:53:06 (1780339986) [ 9440.396942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9441.085718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9445.837547] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9446.373933] LustreError: 324931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9446.402827] LustreError: 324931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9450.479613] LustreError: 324932:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9455.185567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9455.596024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9460.447829] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9466.832413] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9471.452662] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9478.333314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9478.644855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9484.691730] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9490.526863] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9497.987397] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9499.872384] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9501.648964] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9503.297772] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9505.204719] Lustre: Failing over lustre-MDT0000 [ 9505.562812] Lustre: server umount lustre-MDT0000 complete [ 9506.282609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9506.289419] LustreError: 324928:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9506.294434] Lustre: Skipped 1 previous similar message [ 9506.342614] LustreError: 324928:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9511.393615] LustreError: 325891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9511.424156] LustreError: 325891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9516.519241] LustreError: 324931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9516.547605] LustreError: 324931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9521.631263] Lustre: 318799:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780340063/real 1780340063] req@00000000b2a8cf25 x1866821627087104/t0(0) o400->MGC192.168.203.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780340079 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9521.669714] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9524.529774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9526.765603] LustreError: 324931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9526.800366] LustreError: 324931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 9531.875528] LustreError: 318798:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000028b30353 x1866821627091584/t0(0) o250->MGC192.168.203.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9532.238524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9532.321702] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9533.483239] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9536.684828] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9537.542763] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9537.567243] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9537.639930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9543.938952] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9545.789258] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9549.475237] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9551.386561] Lustre: Failing over lustre-MDT0000 [ 9551.696595] Lustre: server umount lustre-MDT0000 complete [ 9552.864862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9552.873574] LustreError: Skipped 1 previous similar message [ 9552.875830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9552.891410] LustreError: 324927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9552.892091] Lustre: Skipped 3 previous similar messages [ 9552.909451] LustreError: 324927:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 9569.247294] Lustre: 318800:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780340110/real 1780340110] req@000000006adae69a x1866821627106816/t0(0) o400->MGC192.168.203.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780340126 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9569.265222] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9570.678244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9578.468131] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8fbbf1a6b36c7d24 [ 9578.488138] Lustre: MGC192.168.203.118@tcp: Connection restored to 0@lo (at 0@lo) [ 9578.494532] Lustre: Skipped 2 previous similar messages [ 9578.789175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9578.856789] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9580.591093] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9583.578825] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9584.126406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9584.165635] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9584.216325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9592.182659] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9593.831414] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9604.582872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9604.583094] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9604.587592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9604.606537] Lustre: Skipped 2 previous similar messages [ 9607.536332] Lustre: server umount lustre-MDT0000 complete [ 9609.702723] LustreError: 324931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9609.730962] LustreError: 324931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 9611.224339] LustreError: 328310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780340168 with bad export cookie 10357137466345684260 [ 9611.225171] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9611.242157] LustreError: 328310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9611.598806] Lustre: server umount lustre-MDT0001 complete [ 9625.989552] Lustre: server umount lustre-OST0000 complete [ 9641.471599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9645.833486] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9653.416630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9653.779770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9653.784254] Lustre: Skipped 1 previous similar message [ 9657.587691] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9662.859539] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9666.931771] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9673.651351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9680.785593] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9686.822547] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9687.347661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9692.642522] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9692.650367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9692.654257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9692.660051] Lustre: Skipped 1 previous similar message [ 9692.675136] Lustre: Skipped 3 previous similar messages [ 9694.519474] Lustre: server umount lustre-OST0000 complete [ 9699.816154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9702.886755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9702.892523] Lustre: Skipped 2 previous similar messages [ 9704.762361] Lustre: server umount lustre-MDT0000 complete [ 9708.001436] LustreError: 330709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9708.043238] LustreError: 330709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9708.427315] LustreError: 330695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780340265 with bad export cookie 10357137466345685009 [ 9708.440620] LustreError: 330695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9708.441869] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9709.221725] Lustre: server umount lustre-MDT0001 complete [ 9720.806499] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 9723.801465] Key type lgssc unregistered [ 9724.088655] LNet: 333888:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9724.094862] LNetError: 333888:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9724.109685] LNet: Removed LNI 192.168.203.118@tcp [ 9725.184610] Key type .llcrypt unregistered [ 9725.191168] Key type ._llcrypt unregistered [ 9752.367609] Key type ._llcrypt registered [ 9752.370931] Key type .llcrypt registered [ 9752.536365] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 9753.801043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9754.006349] alg: No test for adler32 (adler32-zlib) [ 9755.189526] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [ 9755.511361] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 9757.265216] Key type lgssc registered [ 9758.799865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9768.004336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9768.045754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9769.582995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9773.959490] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9780.196648] LustreError: 335391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9780.220854] LustreError: 335391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9782.884403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9783.398189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9788.571145] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9794.853102] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9798.743617] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9805.718932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9806.150890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9812.593248] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9815.417455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9818.728871] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9830.883318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9830.886675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9830.917983] Lustre: Skipped 2 previous similar messages [ 9830.953286] Lustre: Skipped 2 previous similar messages [ 9832.439254] Lustre: server umount lustre-MDT0000 complete [ 9836.017261] LustreError: 335387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9836.052892] LustreError: 335387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9836.886840] LustreError: 336108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780340394 with bad export cookie 2929769589587242522 [ 9836.891596] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9836.928971] LustreError: 336108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9837.549113] Lustre: server umount lustre-MDT0001 complete [ 9852.298529] Lustre: server umount lustre-OST0000 complete [ 9869.019587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9869.569272] LustreError: 338764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9869.654662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9874.917573] LustreError: 338765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9875.007726] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9880.043991] LustreError: 338764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9886.198283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9886.695478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9891.928654] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9899.720832] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9905.823898] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9914.127256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9914.591500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9915.643697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9921.721108] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9928.563586] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9942.500275] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9942.502753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9942.527100] Lustre: Skipped 2 previous similar messages [ 9942.544043] Lustre: Skipped 2 previous similar messages [ 9947.621213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9950.318498] Lustre: server umount lustre-MDT0000 complete [ 9952.738078] LustreError: 338761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9952.786811] LustreError: 338761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9955.862488] LustreError: 338744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780340513 with bad export cookie 2929769589587243635 [ 9955.867747] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9955.875259] LustreError: 338744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9956.659395] Lustre: server umount lustre-MDT0001 complete [ 9972.367903] Lustre: server umount lustre-OST0000 complete [ 9990.474635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9991.026572] LustreError: 342351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9991.154508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9996.267872] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10007.343536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10013.983884] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10020.589835] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10024.855286] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10033.739442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10034.180616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10034.183436] Lustre: Skipped 1 previous similar message [10036.215184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10041.238460] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10047.608992] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10056.672113] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10056.682231] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10056.705046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10056.722538] Lustre: Skipped 2 previous similar messages [10057.705366] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10060.494519] Lustre: server umount lustre-OST0000 complete [10067.937290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10067.972674] Lustre: Skipped 1 previous similar message [10067.997826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10068.019791] Lustre: Skipped 3 previous similar messages [10071.906115] Lustre: server umount lustre-MDT0000 complete [10073.064423] LustreError: 343062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10073.090293] LustreError: 343062:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10075.809765] LustreError: 342333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780340633 with bad export cookie 2929769589587244657 [10075.816123] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10075.830820] LustreError: 342333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10076.272490] Lustre: server umount lustre-MDT0001 complete [10088.573606] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [10091.888222] Key type lgssc unregistered [10092.240390] LNet: 345529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10092.256392] LNetError: 345529:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10092.276355] LNet: Removed LNI 192.168.203.118@tcp [10093.216664] Key type .llcrypt unregistered [10093.218473] Key type ._llcrypt unregistered [10126.165752] Key type ._llcrypt registered [10126.167467] Key type .llcrypt registered [10126.329289] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [10128.416551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10128.465312] alg: No test for adler32 (adler32-zlib) [10129.715093] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [10130.141850] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [10132.087194] Key type lgssc registered [10134.206511] Lustre: Echo OBD driver; http://www.lustre.org/ [10144.522533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10144.567391] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10144.576638] LustreError: 346954:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10144.619541] LustreError: 346954:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10144.631809] LustreError: 346954:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10144.652412] LustreError: 346954:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10144.664937] LustreError: 346954:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10152.975658] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10152.980538] LustreError: 347351:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10152.992762] LustreError: 347351:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10152.998202] LustreError: 347351:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10153.005533] LustreError: 347351:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10153.016370] LustreError: 347351:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10178.923094] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [10182.716177] Key type lgssc unregistered [10183.223673] LNet: 348338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10183.230412] LNetError: 348338:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10183.254744] LNet: Removed LNI 192.168.203.118@tcp [10184.634062] Key type .llcrypt unregistered [10184.636377] Key type ._llcrypt unregistered [10206.158673] Key type ._llcrypt registered [10206.162718] Key type .llcrypt registered [10206.345493] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [10222.203351] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [10224.079730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10224.097931] alg: No test for adler32 (adler32-zlib) [10225.153304] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [10225.408718] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [10227.089084] Key type lgssc registered [10228.278581] Lustre: Echo OBD driver; http://www.lustre.org/ [10268.381167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10268.413972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10269.680769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10269.729922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10269.839642] Lustre: lustre-MDT0000: new disk, initializing [10270.025462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10270.048100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10274.948161] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10287.941147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10288.086573] Lustre: 351464:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10288.143499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10288.148861] Lustre: Skipped 1 previous similar message [10288.252752] Lustre: lustre-MDT0001: new disk, initializing [10288.409635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10288.453705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10288.465641] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10294.627886] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10303.842297] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10308.030878] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10315.758447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10316.155297] Lustre: lustre-OST0000: new disk, initializing [10316.160857] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10316.170261] Lustre: 353205:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10316.304955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10324.046821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10324.071546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10324.202865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10324.217916] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10333.189710] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10338.766298] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10339.042518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10344.681854] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10344.984939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10350.063961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10350.064589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10350.095551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10350.105317] Lustre: Skipped 1 previous similar message [10355.177188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10360.290965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10360.294488] Lustre: Skipped 2 previous similar messages [10361.311147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10361.680339] Lustre: server umount lustre-OST0000 complete [10370.528490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10370.552851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10370.554924] Lustre: Skipped 1 previous similar message [10370.559695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10370.600176] LustreError: Skipped 1 previous similar message [10372.602019] Lustre: server umount lustre-MDT0000 complete [10376.925097] LustreError: 351454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780340934 with bad export cookie 9235974691210774814 [10376.929107] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10376.941102] LustreError: 351454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10377.307588] Lustre: server umount lustre-MDT0001 complete [10387.735715] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10389.761766] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 15:09:04 (1780340944) [10397.913778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10398.671638] LustreError: 355522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10398.708117] LustreError: 355522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10398.820992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10403.812100] LustreError: 355523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10404.240751] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10408.938116] LustreError: 355522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10413.964236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10414.061812] LustreError: 355523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10414.507563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10420.246953] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10426.898539] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10431.012715] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10438.835043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10439.137754] LustreError: 357117:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10439.434468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10447.411974] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10454.086740] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10461.543816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10461.813636] Lustre: lustre-OST0001: new disk, initializing [10461.818543] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10461.827562] Lustre: 358021:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10461.976232] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10469.022928] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10469.539042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10469.562233] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10469.701126] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10478.144151] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10485.216739] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10485.219799] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10485.224151] LustreError: Skipped 1 previous similar message [10485.227995] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10485.234871] Lustre: Skipped 2 previous similar messages [10485.247409] Lustre: Skipped 2 previous similar messages [10489.106816] Lustre: server umount lustre-OST0001 complete [10490.344357] LustreError: 357306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10490.378597] LustreError: 357306:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10510.304266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10510.306714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10510.327727] LustreError: Skipped 1 previous similar message [10510.331356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10510.336881] Lustre: Skipped 1 previous similar message [10510.343435] Lustre: Skipped 1 previous similar message [10510.821263] LustreError: 357306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10510.850208] LustreError: 357306:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [10524.127453] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10524.429550] Lustre: server umount lustre-OST0000 complete [10531.306246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10531.309052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10531.338488] Lustre: Skipped 1 previous similar message [10531.355920] Lustre: Skipped 6 previous similar messages [10535.936836] Lustre: server umount lustre-MDT0000 complete [10539.795795] LustreError: 358020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780341097 with bad export cookie 9235974691210775703 [10539.800386] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10539.826895] LustreError: 358020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10554.335210] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10555.175777] Lustre: server umount lustre-MDT0001 complete [10569.485712] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [10572.960707] Key type lgssc unregistered [10573.399499] LNet: 359880:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10573.409973] LNetError: 359880:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10574.443345] LNet: Removed LNI 192.168.203.118@tcp [10575.559788] Key type .llcrypt unregistered [10575.563830] Key type ._llcrypt unregistered [10634.896244] Key type ._llcrypt registered [10634.900631] Key type .llcrypt registered [10635.034727] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [10636.870338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10636.924338] alg: No test for adler32 (adler32-zlib) [10638.167567] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [10638.409267] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [10640.151174] Key type lgssc registered [10641.671604] Lustre: Echo OBD driver; http://www.lustre.org/ [10650.846717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10650.887600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10652.317411] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10652.341718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10652.651660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10657.769259] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10668.314644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10668.440195] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10668.463969] Lustre: 362662:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10668.721770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10673.125914] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10679.185171] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10684.080522] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10689.507122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10689.512934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10689.520606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10690.017879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10690.049940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10690.060736] Lustre: Skipped 1 previous similar message [10692.465357] Lustre: server umount lustre-MDT0000 complete [10696.517506] LustreError: 362655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780341254 with bad export cookie 11076594372858530246 [10696.520558] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10696.533745] LustreError: 362655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10697.168287] Lustre: server umount lustre-MDT0001 complete [10707.867142] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 15:14:22 (1780341262) [10715.127330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10715.760127] LustreError: 364952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10715.781302] LustreError: 364952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10715.873498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10721.256471] LustreError: 364953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10721.351458] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10726.373657] LustreError: 364952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10729.881925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10730.229735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10735.034654] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10740.684520] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10744.719832] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10750.967369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10751.167625] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10751.344391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10757.823469] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10763.873715] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10771.781704] Lustre: Permanently deactivating lustre-OST0000 [10771.806030] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10781.755700] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10783.528335] LustreError: 367646:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10783.546236] LustreError: 367646:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [10783.559673] LustreError: 367646:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10786.597943] LustreError: 367694:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [10786.606343] LustreError: 367694:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10794.858994] Lustre: server umount lustre-OST0000 complete [10798.271397] LustreError: 367944:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [10798.279141] LustreError: 367944:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10802.146731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10802.159289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10802.171860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10804.705187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10804.706756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10804.735908] Lustre: Skipped 1 previous similar message [10809.825305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10809.840059] Lustre: Skipped 1 previous similar message [10810.914940] Lustre: server umount lustre-MDT0000 complete [10814.305103] LustreError: 365666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780341371 with bad export cookie 11076594372858530834 [10814.316151] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10814.322353] LustreError: 365666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10823.884380] Lustre: server umount lustre-MDT0001 complete [10832.762414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10837.354483] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10840.691583] LustreError: 369086:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10847.542926] Lustre: 369426:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.118@tcp [10849.601802] Lustre: server umount MGS complete [10861.157542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10861.670514] LustreError: 370150:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10861.705435] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10861.711435] Lustre: Skipped 1 previous similar message [10861.784409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10866.445235] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10867.171224] LustreError: 370151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10874.522272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10874.859321] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10878.810397] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10883.964686] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10887.451836] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10889.029792] Lustre: Permanently reactivating lustre-OST0000 [10889.055660] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10889.062322] Lustre: Skipped 2 previous similar messages [10916.319220] LustreError: 371703:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10922.596362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10922.906260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10922.912278] Lustre: Skipped 1 previous similar message [10928.806692] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10933.656920] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10941.412252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10941.419900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10941.451689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10941.463575] Lustre: Skipped 1 previous similar message [10945.132060] Lustre: server umount lustre-OST0000 complete [10951.653804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10951.669457] Lustre: Skipped 1 previous similar message [10951.688966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10955.878919] Lustre: server umount lustre-MDT0000 complete [10956.782404] LustreError: 370145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10956.812799] LustreError: 370145:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10960.483132] LustreError: 370130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780341518 with bad export cookie 11076594372858531884 [10960.505623] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10961.044563] Lustre: server umount lustre-MDT0001 complete [10973.766214] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [10976.884545] Key type lgssc unregistered [10977.287531] LNet: 374155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10977.298811] LNetError: 374155:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10977.312911] LNet: Removed LNI 192.168.203.118@tcp [10978.235773] Key type .llcrypt unregistered [10978.238235] Key type ._llcrypt unregistered [10996.892866] Key type ._llcrypt registered [10996.904625] Key type .llcrypt registered [10997.039982] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [11011.125977] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [11012.138242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11012.233200] alg: No test for adler32 (adler32-zlib) [11013.290890] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [11013.553176] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [11015.303357] Key type lgssc registered [11016.519127] Lustre: Echo OBD driver; http://www.lustre.org/ [11047.881791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11047.926145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11049.221188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11049.268855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11049.354946] Lustre: lustre-MDT0000: new disk, initializing [11049.474843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11049.503913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11053.236748] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11062.548649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11062.677947] Lustre: 377280:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11062.738731] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11062.741598] Lustre: Skipped 1 previous similar message [11062.831447] Lustre: lustre-MDT0001: new disk, initializing [11062.917704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11062.952722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11062.962820] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11066.991322] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11073.472957] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11076.820686] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11082.717722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11082.947456] Lustre: lustre-OST0000: new disk, initializing [11082.950290] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11082.953820] Lustre: 379019:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11083.032288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11088.769820] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11093.074961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11093.086451] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11093.217573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11096.280518] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11100.430800] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11100.625254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11104.591655] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11104.772832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11108.838404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11108.846702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11108.870711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11108.877912] Lustre: Skipped 1 previous similar message [11112.570548] Lustre: server umount lustre-OST0000 complete [11119.079461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11119.089124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11119.108019] Lustre: Skipped 2 previous similar messages [11119.120778] Lustre: Skipped 1 previous similar message [11121.839850] Lustre: server umount lustre-MDT0000 complete [11124.941184] LustreError: 377271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780341682 with bad export cookie 15666765083704372627 [11124.942284] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11124.956924] LustreError: 377271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11125.288919] Lustre: server umount lustre-MDT0001 complete [11132.431845] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11134.105585] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 15:21:29 (1780341689) [11140.383221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11140.946712] LustreError: 381331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11140.983426] LustreError: 381331:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11141.054579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11145.228667] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11146.211712] LustreError: 381332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11146.241708] LustreError: 381332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11151.333407] LustreError: 381331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11153.001918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11153.379313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11157.828183] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11162.589221] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11166.107644] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11172.860784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11173.152356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11178.725689] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11183.645747] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11185.433516] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11187.607353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11193.825644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11193.831534] LustreError: Skipped 1 previous similar message [11193.838372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11193.850683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11195.876909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11195.879667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11195.885773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11195.910940] Lustre: Skipped 1 previous similar message [11197.905383] Lustre: server umount lustre-OST0000 complete [11200.995714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11200.998418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11201.012396] Lustre: Skipped 1 previous similar message [11201.027839] Lustre: Skipped 1 previous similar message [11211.232929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11211.236147] Lustre: Skipped 3 previous similar messages [11215.327232] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11215.451198] Lustre: server umount lustre-MDT0000 complete [11216.352193] LustreError: 381331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11216.376026] LustreError: 381331:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11218.697205] LustreError: 381313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780341776 with bad export cookie 15666765083704373509 [11218.701884] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11218.710182] LustreError: 381313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11219.014873] Lustre: server umount lustre-MDT0001 complete [11227.781755] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [11230.261819] Key type lgssc unregistered [11230.484560] LNet: 384699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11230.488676] LNetError: 384699:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11230.499357] LNet: Removed LNI 192.168.203.118@tcp [11231.164253] Key type .llcrypt unregistered [11231.166234] Key type ._llcrypt unregistered [11252.138805] Key type ._llcrypt registered [11252.140519] Key type .llcrypt registered [11252.227604] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [11252.433598] Key type .llcrypt unregistered [11252.439405] Key type ._llcrypt unregistered [11268.094570] Key type ._llcrypt registered [11268.096856] Key type .llcrypt registered [11268.198586] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [11269.376484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11269.411276] alg: No test for adler32 (adler32-zlib) [11270.419371] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [11270.607301] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [11272.311615] Key type lgssc registered [11273.392507] Lustre: Echo OBD driver; http://www.lustre.org/ [11281.114213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11281.135475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11282.619075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11286.769225] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11293.153720] LustreError: 386883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11293.167837] LustreError: 386883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11294.520039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11294.841574] LustreError: 386884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11294.874746] LustreError: 386884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11294.892409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11300.320754] LustreError: 386883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11300.344154] LustreError: 386883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11300.855133] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11305.441686] LustreError: 386884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11305.472519] LustreError: 386884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11306.540971] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11307.511147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11312.609686] LustreError: 386883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11312.634782] LustreError: 386883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11313.794039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11314.156116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11318.053868] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11327.229620] Lustre: server umount lustre-OST0000 complete [11333.093727] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11333.104729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11333.126438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11334.628326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11334.636887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11334.661165] Lustre: Skipped 1 previous similar message [11339.747329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11339.758578] Lustre: Skipped 1 previous similar message [11340.814946] Lustre: server umount lustre-MDT0000 complete [11343.850794] LustreError: 386866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780341901 with bad export cookie 10206981382819059284 [11343.854847] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11343.861295] LustreError: 386866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11354.234288] Lustre: server umount lustre-MDT0001 complete [11363.030297] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [11365.871723] Key type lgssc unregistered [11366.169914] LNet: 389869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11366.179161] LNetError: 389869:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11367.206953] LNet: Removed LNI 192.168.203.118@tcp [11368.038683] Key type .llcrypt unregistered [11368.040630] Key type ._llcrypt unregistered [11390.200239] Key type ._llcrypt registered [11390.204644] Key type .llcrypt registered [11390.310385] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [11391.523661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11391.731453] alg: No test for adler32 (adler32-zlib) [11392.800286] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [11393.153094] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [11394.815619] Key type lgssc registered [11396.020723] Lustre: Echo OBD driver; http://www.lustre.org/ [11402.218829] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11402.235073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11417.759316] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11422.879319] LustreError: 391243:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.118@tcp: failed processing log, type 4: rc = -110 [11449.568697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11455.981586] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11460.589663] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11465.926916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11466.291649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11466.343807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11469.433102] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11471.332041] LustreError: 391269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11471.349690] LustreError: 391269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11476.257031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11476.450690] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11476.582667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11479.663504] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11484.651602] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11487.898437] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11494.370400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11494.372858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11494.387808] Lustre: Skipped 1 previous similar message [11496.325711] Lustre: server umount lustre-OST0000 complete [11499.489971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11499.500797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11499.511263] Lustre: Skipped 1 previous similar message [11499.515477] Lustre: Skipped 1 previous similar message [11504.608199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11504.617859] Lustre: Skipped 2 previous similar messages [11505.668271] Lustre: server umount lustre-MDT0000 complete [11508.701420] LustreError: 392890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780342066 with bad export cookie 14860372124275638195 [11508.707825] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11508.713762] LustreError: 392890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11508.996438] Lustre: server umount lustre-MDT0001 complete [11517.959838] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [11520.343729] Key type lgssc unregistered [11520.646686] LNet: 394455:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11520.654779] LNetError: 394455:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11521.705439] LNet: Removed LNI 192.168.203.118@tcp [11522.451074] Key type .llcrypt unregistered [11522.453675] Key type ._llcrypt unregistered [11542.074642] Key type ._llcrypt registered [11542.078402] Key type .llcrypt registered [11542.191467] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [11543.182050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11543.219623] alg: No test for adler32 (adler32-zlib) [11544.282551] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [11544.489974] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [11546.159223] Key type lgssc registered [11546.966757] Lustre: Echo OBD driver; http://www.lustre.org/ [11552.947944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11552.963295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11554.370655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11557.750987] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11563.494579] LustreError: 395833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11563.507810] LustreError: 395833:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11564.542494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11564.942175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11568.537516] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11573.220738] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11576.497026] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11581.789419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11582.031243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11586.843382] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11591.207635] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11599.409411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11604.451081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11604.456348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11604.463949] Lustre: Skipped 2 previous similar messages [11604.472238] Lustre: Skipped 1 previous similar message [11607.100766] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.18@tcp (stopping) [11607.106858] Lustre: Skipped 1 previous similar message [11609.571991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11609.580120] Lustre: Skipped 2 previous similar messages [11610.293277] Lustre: server umount lustre-MDT0000 complete [11612.007736] LustreError: 395815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.18@tcp arrived at 1780342169 with bad export cookie 16049033671530316404 [11612.015218] LustreError: 395815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11613.529870] LustreError: 397448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780342171 with bad export cookie 16049033671530315739 [11613.536046] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11613.543672] LustreError: 397448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11614.688249] LustreError: 395833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11614.689353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11614.701934] LustreError: 395833:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11614.737489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11618.720667] LustreError: 395828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11618.735078] LustreError: 395828:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11619.729240] Lustre: server umount lustre-OST0000 complete [11623.841522] LustreError: 395833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11623.856184] LustreError: 395833:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11625.133658] Lustre: server umount lustre-MDT0001 complete [11633.293681] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [11635.309297] Key type lgssc unregistered [11635.587456] LNet: 399179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11635.599458] LNetError: 399179:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11635.615952] LNet: Removed LNI 192.168.203.118@tcp [11636.323977] Key type .llcrypt unregistered [11636.325689] Key type ._llcrypt unregistered [11654.677845] Key type ._llcrypt registered [11654.680581] Key type .llcrypt registered [11654.754423] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [11655.443285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11655.485537] alg: No test for adler32 (adler32-zlib) [11656.496574] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [11656.690235] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [11658.359140] Key type lgssc registered [11659.261749] Lustre: Echo OBD driver; http://www.lustre.org/ [11665.946449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11665.974472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11667.338606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11670.779343] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11676.642860] LustreError: 400559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11676.673208] LustreError: 400559:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11677.922972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11678.308694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11681.916528] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11686.064230] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11688.584784] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11692.932894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11693.189602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11697.649139] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11701.816245] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11711.664608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11717.091788] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11717.114353] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11717.122189] Lustre: Skipped 1 previous similar message [11718.716114] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.18@tcp (stopping) [11722.209835] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11722.218833] Lustre: Skipped 1 previous similar message [11727.329931] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11727.334952] Lustre: Skipped 1 previous similar message [11729.887284] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11730.151740] Lustre: server umount lustre-MDT0001 complete [11732.455770] LustreError: 401591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11732.483057] LustreError: 401591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11737.576339] LustreError: 400560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11737.577032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11737.577285] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11737.606358] Lustre: Skipped 1 previous similar message [11737.611067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11737.614046] Lustre: Skipped 1 previous similar message [11741.415424] Lustre: server umount lustre-OST0000 complete [11742.690446] LustreError: 400559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11744.797931] Lustre: server umount lustre-MDT0000 complete [11755.802160] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [11757.785550] Key type lgssc unregistered [11757.987471] LNet: 403908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11757.998460] LNetError: 403908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11758.014096] LNet: Removed LNI 192.168.203.118@tcp [11758.527091] Key type .llcrypt unregistered [11758.531105] Key type ._llcrypt unregistered [11776.258692] Key type ._llcrypt registered [11776.260960] Key type .llcrypt registered [11776.342353] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [11787.780612] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [11788.648272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11788.665168] alg: No test for adler32 (adler32-zlib) [11789.664917] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [11789.859943] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [11791.503236] Key type lgssc registered [11792.515634] Lustre: Echo OBD driver; http://www.lustre.org/ [11821.084301] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [11833.680445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11833.708614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11834.897926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11834.924844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11834.996843] Lustre: lustre-MDT0000: new disk, initializing [11835.089994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11835.112301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11838.166648] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11849.815893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11850.068277] Lustre: lustre-OST0000: new disk, initializing [11850.072300] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11850.075494] Lustre: Skipped 1 previous similar message [11850.081437] Lustre: 408850:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11850.152912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11850.434624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11850.441874] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11850.467773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11854.390603] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11860.331772] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11871.391655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11871.532441] Lustre: 407848:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11871.599193] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11871.686879] Lustre: lustre-MDT0001: new disk, initializing [11871.818768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11871.892424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11871.904637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11875.219324] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11882.610187] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11882.788747] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11886.059297] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11886.210905] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11892.292237] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.18@tcp (stopping) [11893.224634] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11893.226221] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11893.238712] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11893.259027] Lustre: Skipped 1 previous similar message [11895.858302] Lustre: server umount lustre-MDT0001 complete [11903.462909] LustreError: 407856:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11903.482764] LustreError: 407856:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11908.581243] LustreError: 407857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11908.603143] LustreError: 407857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11912.161715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11912.170308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11912.181417] Lustre: Skipped 1 previous similar message [11912.184904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11913.701327] LustreError: 407861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11917.056802] Lustre: server umount lustre-OST0000 complete [11918.819507] LustreError: 407862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11928.241780] Lustre: server umount lustre-MDT0000 complete [11937.481149] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [11939.445731] Key type lgssc unregistered [11939.690854] LNet: 412411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11939.699616] LNetError: 412411:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11939.713812] LNet: Removed LNI 192.168.203.118@tcp [11940.299806] Key type .llcrypt unregistered [11940.304168] Key type ._llcrypt unregistered [11968.385820] Key type ._llcrypt registered [11968.387878] Key type .llcrypt registered [11968.463139] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [11968.682532] Key type .llcrypt unregistered [11968.692956] Key type ._llcrypt unregistered [11980.587435] Key type ._llcrypt registered [11980.590106] Key type .llcrypt registered [11980.653457] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [11981.373517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11981.392261] alg: No test for adler32 (adler32-zlib) [11982.364557] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [11982.594103] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [11984.239164] Key type lgssc registered [11984.984845] Lustre: Echo OBD driver; http://www.lustre.org/ [12004.172856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12004.196934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12005.388659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12005.415193] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12005.496318] Lustre: lustre-MDT0000: new disk, initializing [12005.564470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12005.582133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12009.053344] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [12017.669525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12017.755397] Lustre: 415732:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12017.805285] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12017.810773] Lustre: Skipped 1 previous similar message [12017.877420] Lustre: lustre-MDT0001: new disk, initializing [12018.005456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12018.050297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12018.060764] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12021.161968] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [12027.454539] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12030.205821] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12035.620220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12035.821827] Lustre: lustre-OST0000: new disk, initializing [12035.829389] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12035.834692] Lustre: 417474:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12035.919468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12040.928873] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [12044.850443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12044.868199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12044.985772] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12047.424533] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12053.985271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12053.992298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12054.004671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12055.519951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12055.527530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12055.541795] Lustre: Skipped 1 previous similar message [12058.563507] Lustre: server umount lustre-MDT0000 complete [12061.974078] LustreError: 417473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780342619 with bad export cookie 8828671251408097644 [12061.974191] LustreError: 416511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780342619 with bad export cookie 8828671251408097644 [12061.979636] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12061.981513] LustreError: 417473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12062.275571] Lustre: server umount lustre-MDT0001 complete [12074.704776] Lustre: server umount lustre-OST0000 complete [12082.530405] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 15:37:18 (1780342638) [12088.768836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12089.076784] LustreError: 419387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12089.096773] LustreError: 419387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12089.159110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12092.430480] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [12094.432392] LustreError: 419388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12098.539900] LustreError: 419387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12098.619763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12098.930281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12102.957090] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [12107.226606] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12110.138419] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12127.471538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12127.595974] Lustre: Found index 0 for lustre-OST0000, updating log [12127.712237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12131.384737] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [12133.870524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12135.093960] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12138.978796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12138.980111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12138.989064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12139.000042] Lustre: Skipped 2 previous similar messages [12140.206445] Lustre: server umount lustre-OST0000 complete [12155.363635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12155.366418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12155.367698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12155.367704] Lustre: Skipped 1 previous similar message [12155.374027] Lustre: Skipped 1 previous similar message [12159.457133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12159.465973] Lustre: Skipped 2 previous similar messages [12160.547926] Lustre: server umount lustre-MDT0000 complete [12163.212394] LustreError: 419369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780342720 with bad export cookie 8828671251408099030 [12163.219175] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12163.220296] LustreError: 419369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12163.465923] Lustre: server umount lustre-MDT0001 complete [12169.441728] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 15:38:45 (1780342725) [12195.615428] LNet: 1 local NIs in recovery (showing 1): 192.168.203.118@tcp [12786.858106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12787.190662] LustreError: 425454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12787.257338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12791.242578] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [12792.292973] LustreError: 425455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12797.415936] LustreError: 425454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12799.004673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12799.330547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12803.427702] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [12809.123452] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12812.751474] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12832.175983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12832.405974] Lustre: Found index 0 for lustre-OST0000, updating log [12832.783903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12834.798049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [12837.735961] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [12843.167428] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12846.379099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12846.413366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12846.426506] Lustre: Skipped 2 previous similar messages [12847.904878] LNet: 1 local NIs in recovery (showing 1): 192.168.203.118@tcp [12851.547584] Lustre: server umount lustre-OST0000 complete [12855.507658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12855.523440] Lustre: Skipped 1 previous similar message [12855.523730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12857.633136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12857.649948] Lustre: Skipped 4 previous similar messages [12861.729893] LustreError: 425454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12861.758131] LustreError: 425454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12863.777890] Lustre: server umount lustre-MDT0000 complete [12865.835700] LustreError: 426165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12865.849925] LustreError: 426165:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [12867.111075] LustreError: 425435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343424 with bad export cookie 8828671251408100101 [12867.118740] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12867.555589] Lustre: server umount lustre-MDT0001 complete [12894.774497] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [12902.792987] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [12932.275318] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 15:51:27 (1780343487) [12947.266512] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [12949.562928] Key type lgssc unregistered [12949.802754] LNet: 431670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12949.807931] LNetError: 431670:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13047.079698] LNet: Removed LNI 192.168.203.118@tcp [13047.903526] Key type .llcrypt unregistered [13047.905351] Key type ._llcrypt unregistered [13060.335852] Key type ._llcrypt registered [13060.339804] Key type .llcrypt registered [13060.412739] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [13061.159114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13061.167970] alg: No test for adler32 (adler32-zlib) [13062.135812] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [13062.261984] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [13063.887252] Key type lgssc registered [13064.705766] Lustre: Echo OBD driver; http://www.lustre.org/ [13068.217743] LNet: Added LNI 192.168.203.118@tcp100 [8/256/0/180] [13069.005331] LNet: Added LNI 192.168.203.118@tcp101 [8/256/0/180] [13069.683872] LNet: Added LNI 192.168.203.118@tcp102 [8/256/0/180] [13070.359970] LNet: Added LNI 192.168.203.118@tcp103 [8/256/0/180] [13071.061794] LNet: Added LNI 192.168.203.118@tcp104 [8/256/0/180] [13071.758912] LNet: Added LNI 192.168.203.118@tcp105 [8/256/0/180] [13072.457926] LNet: Added LNI 192.168.203.118@tcp106 [8/256/0/180] [13073.248624] LNet: Added LNI 192.168.203.118@tcp107 [8/256/0/180] [13074.028925] LNet: Added LNI 192.168.203.118@tcp108 [8/256/0/180] [13074.708710] LNet: Added LNI 192.168.203.118@tcp109 [8/256/0/180] [13075.402243] LNet: Added LNI 192.168.203.118@tcp110 [8/256/0/180] [13076.088716] LNet: Added LNI 192.168.203.118@tcp111 [8/256/0/180] [13076.714620] LNet: Added LNI 192.168.203.118@tcp112 [8/256/0/180] [13077.358450] LNet: Added LNI 192.168.203.118@tcp113 [8/256/0/180] [13078.107344] LNet: Added LNI 192.168.203.118@tcp114 [8/256/0/180] [13078.733928] LNet: Added LNI 192.168.203.118@tcp115 [8/256/0/180] [13079.371452] LNet: Added LNI 192.168.203.118@tcp116 [8/256/0/180] [13080.172689] LNet: Added LNI 192.168.203.118@tcp117 [8/256/0/180] [13080.942456] LNet: Added LNI 192.168.203.118@tcp118 [8/256/0/180] [13081.711826] LNet: Added LNI 192.168.203.118@tcp119 [8/256/0/180] [13082.450508] LNet: Added LNI 192.168.203.118@tcp120 [8/256/0/180] [13083.227269] LNet: Added LNI 192.168.203.118@tcp121 [8/256/0/180] [13083.963747] LNet: Added LNI 192.168.203.118@tcp122 [8/256/0/180] [13084.710810] LNet: Added LNI 192.168.203.118@tcp123 [8/256/0/180] [13085.406152] LNet: Added LNI 192.168.203.118@tcp124 [8/256/0/180] [13086.135749] LNet: Added LNI 192.168.203.118@tcp125 [8/256/0/180] [13086.839323] LNet: Added LNI 192.168.203.118@tcp126 [8/256/0/180] [13087.591546] LNet: Added LNI 192.168.203.118@tcp127 [8/256/0/180] [13088.193872] LNet: Added LNI 192.168.203.118@tcp128 [8/256/0/180] [13088.890857] LNet: Added LNI 192.168.203.118@tcp129 [8/256/0/180] [13089.595894] LNet: Added LNI 192.168.203.118@tcp130 [8/256/0/180] [13090.332800] LNet: Added LNI 192.168.203.118@tcp131 [8/256/0/180] [13090.973453] LNet: Added LNI 192.168.203.118@tcp132 [8/256/0/180] [13091.715720] LNet: Added LNI 192.168.203.118@tcp133 [8/256/0/180] [13092.430957] LNet: Added LNI 192.168.203.118@tcp134 [8/256/0/180] [13093.158481] LNet: Added LNI 192.168.203.118@tcp135 [8/256/0/180] [13098.593223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13098.616684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13099.697639] Lustre: 434663:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.118@tcp: can't add NID 192.168.203.118@tcp130: rc = -75 [13099.755848] LNetError: 434687:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.118 [13099.771665] LNetError: Protocol error connecting to 192.168.203.118@tcp121 at host 192.168.203.118:988: is it running a compatible version of Lustre? [13099.777959] LNetError: Skipped 4 previous similar messages [13099.878907] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13099.897858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13099.946359] Lustre: lustre-MDT0000: new disk, initializing [13100.016105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13100.039631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13100.192326] LNet: 7 local NIs in recovery (showing 7): 192.168.203.118@tcp126, 192.168.203.118@tcp127, 192.168.203.118@tcp128, 192.168.203.118@tcp129, 192.168.203.118@tcp121, 192.168.203.118@tcp125, 192.168.203.118@tcp115 [13102.241232] LNetError: 432120:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.118 [13102.254121] LNetError: 432120:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 21 previous similar messages [13102.259030] LNetError: Protocol error connecting to 192.168.203.118@tcp127 at host 192.168.203.118:988: is it running a compatible version of Lustre? [13102.268049] LNetError: Skipped 17 previous similar messages [13102.855948] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [13106.339855] LNetError: 434683:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.118 [13106.351769] LNetError: 434683:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [13106.360227] LNetError: Protocol error connecting to 192.168.203.118@tcp127 at host 192.168.203.118:988: is it running a compatible version of Lustre? [13110.880093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13110.967962] Lustre: 434680:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13111.012909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13111.018450] Lustre: Skipped 1 previous similar message [13111.079357] Lustre: lustre-MDT0001: new disk, initializing [13111.110620] Lustre: 435467:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.118@tcp: can't add NID 192.168.203.118@tcp130: rc = -75 [13111.117960] Lustre: 435467:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13111.184886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13111.211410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13111.216468] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13114.066807] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [13114.528375] LNetError: 434688:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.118 [13114.540283] LNetError: Protocol error connecting to 192.168.203.118@tcp127 at host 192.168.203.118:988: is it running a compatible version of Lustre? [13116.958723] Lustre: 435845:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.118@tcp: can't add NID 192.168.203.118@tcp130: rc = -75 [13116.966659] Lustre: 435845:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13119.584304] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13122.366966] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13127.138118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13127.338282] Lustre: lustre-OST0000: new disk, initializing [13127.341594] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13127.346600] Lustre: 436430:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13127.402233] Lustre: 436435:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.118@tcp: can't add NID 192.168.203.118@tcp130: rc = -75 [13127.406518] Lustre: 436435:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [13127.440652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13129.889681] LNetError: 434689:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.118 [13129.899815] LNetError: Protocol error connecting to 192.168.203.118@tcp116 at host 192.168.203.118:988: is it running a compatible version of Lustre? [13131.434335] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [13133.835792] Lustre: 436888:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.118@tcp: can't add NID 192.168.203.118@tcp130: rc = -75 [13133.844567] Lustre: 436888:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13133.882668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13133.888546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13133.982249] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13136.462806] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13138.080948] LNetError: 434683:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.118 [13138.087348] LNetError: 434683:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [13138.095211] LNetError: Protocol error connecting to 192.168.203.118@tcp123 at host 192.168.203.118:988: is it running a compatible version of Lustre? [13138.104058] LNetError: Skipped 7 previous similar messages [13144.547356] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13144.552150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13144.560099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13144.570220] Lustre: Skipped 1 previous similar message [13149.670924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13149.676521] Lustre: Skipped 1 previous similar message [13154.464564] LNetError: 432122:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.118 [13154.474921] LNetError: 432122:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 13 previous similar messages [13154.482488] LNetError: Protocol error connecting to 192.168.203.118@tcp108 at host 192.168.203.118:988: is it running a compatible version of Lustre? [13154.489362] LNetError: Skipped 13 previous similar messages [13154.789582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13154.794760] Lustre: Skipped 1 previous similar message [13155.295182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13155.429324] Lustre: server umount lustre-OST0000 complete [13159.583155] LNet: 1 local NIs in recovery (showing 1): 192.168.203.118@tcp127 [13159.906487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13159.919786] Lustre: Skipped 1 previous similar message [13159.928651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13159.934227] Lustre: Skipped 1 previous similar message [13164.262867] Lustre: server umount lustre-MDT0000 complete [13166.631402] LustreError: 435466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343724 with bad export cookie 16424631914013374851 [13166.635735] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13166.646140] LustreError: 435466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13166.988362] Lustre: server umount lustre-MDT0001 complete [13175.322122] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [13177.595921] Key type lgssc unregistered [13177.967885] LNet: 438211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13177.977657] LNetError: 438211:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13177.990249] LNet: Removed LNI 192.168.203.118@tcp [13179.039192] LNet: Removed LNI 192.168.203.118@tcp100 [13179.042415] LNetError: 438211:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13179.050383] LNetError: 438211:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13179.056899] LNet: Removed LNI 192.168.203.118@tcp101 [13180.063162] LNet: Removed LNI 192.168.203.118@tcp102 [13180.073770] LNetError: 438211:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13180.084092] LNetError: 438211:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13181.151915] LNet: Removed LNI 192.168.203.118@tcp103 [13182.175526] LNet: Removed LNI 192.168.203.118@tcp104 [13182.180842] LNetError: 438211:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13182.205221] LNetError: 438211:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13182.222592] LNet: Removed LNI 192.168.203.118@tcp105 [13183.263189] LNet: Removed LNI 192.168.203.118@tcp106 [13184.287167] LNet: Removed LNI 192.168.203.118@tcp107 [13185.311160] LNet: Removed LNI 192.168.203.118@tcp108 [13186.335290] LNet: Removed LNI 192.168.203.118@tcp109 [13186.339031] LNetError: 438211:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13186.354183] LNetError: 438211:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13186.361880] LNet: Removed LNI 192.168.203.118@tcp110 [13187.423171] LNet: Removed LNI 192.168.203.118@tcp111 [13188.447164] LNet: Removed LNI 192.168.203.118@tcp112 [13189.472255] LNet: Removed LNI 192.168.203.118@tcp113 [13190.499070] LNet: Removed LNI 192.168.203.118@tcp114 [13191.519249] LNet: Removed LNI 192.168.203.118@tcp115 [13192.543454] LNet: Removed LNI 192.168.203.118@tcp116 [13193.567227] LNet: Removed LNI 192.168.203.118@tcp117 [13194.591910] LNet: Removed LNI 192.168.203.118@tcp118 [13194.599536] LNetError: 438211:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13194.607469] LNetError: 438211:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13194.615377] LNet: Removed LNI 192.168.203.118@tcp119 [13195.679259] LNet: Removed LNI 192.168.203.118@tcp120 [13196.703353] LNet: Removed LNI 192.168.203.118@tcp121 [13197.728480] LNet: Removed LNI 192.168.203.118@tcp122 [13198.751368] LNet: Removed LNI 192.168.203.118@tcp123 [13199.776832] LNet: Removed LNI 192.168.203.118@tcp124 [13200.799514] LNet: Removed LNI 192.168.203.118@tcp125 [13201.823702] LNet: Removed LNI 192.168.203.118@tcp126 [13201.828939] LNet: Removed LNI 192.168.203.118@tcp127 [13202.849585] LNet: Removed LNI 192.168.203.118@tcp128 [13203.874932] LNet: Removed LNI 192.168.203.118@tcp129 [13203.877401] LNet: Removed LNI 192.168.203.118@tcp130 [13203.881053] LNet: Removed LNI 192.168.203.118@tcp131 [13203.885156] LNet: Removed LNI 192.168.203.118@tcp132 [13203.890529] LNet: Removed LNI 192.168.203.118@tcp133 [13203.896242] LNet: Removed LNI 192.168.203.118@tcp134 [13203.915743] LNet: Removed LNI 192.168.203.118@tcp135 [13204.499659] Key type .llcrypt unregistered [13204.503073] Key type ._llcrypt unregistered [13222.106368] Key type ._llcrypt registered [13222.108459] Key type .llcrypt registered [13222.214361] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [13233.825679] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [13234.473677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13234.552033] alg: No test for adler32 (adler32-zlib) [13235.545619] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [13235.821470] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [13237.503188] Key type lgssc registered [13238.546527] Lustre: Echo OBD driver; http://www.lustre.org/ [13265.797491] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 15:57:01 (1780343821) [13278.879919] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [13281.167937] Key type lgssc unregistered [13281.384140] LNet: 442223:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13281.389324] LNetError: 442223:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13281.402416] LNet: Removed LNI 192.168.203.118@tcp [13282.098536] Key type .llcrypt unregistered [13282.101197] Key type ._llcrypt unregistered [13296.013824] Key type ._llcrypt registered [13296.018937] Key type .llcrypt registered [13296.103264] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [13296.989099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13297.009085] alg: No test for adler32 (adler32-zlib) [13298.061535] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [13298.251726] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [13299.927170] Key type lgssc registered [13300.767253] Lustre: Echo OBD driver; http://www.lustre.org/ [13307.249416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13307.275134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13308.431494] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13308.471909] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13308.536830] Lustre: lustre-MDT0000: new disk, initializing [13308.630238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13308.646061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13311.513539] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [13318.499625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13318.593788] Lustre: 443412:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13318.634982] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13318.639186] Lustre: Skipped 1 previous similar message [13318.702672] Lustre: lustre-MDT0001: new disk, initializing [13318.767681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13318.800328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13318.815658] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13321.656882] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [13327.468977] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13329.913594] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13334.802537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13334.960895] Lustre: lustre-OST0000: new disk, initializing [13334.963896] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13334.972111] Lustre: 445152:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13335.044858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13339.445922] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [13340.733547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13340.741964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13340.841642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13345.136665] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13347.663600] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing lnet_if_list [13351.370995] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13351.411123] LNet: Added LNI 192.168.203.118@tcp10 [8/256/0/180] [13355.927113] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13355.957794] LNet: Added LNI 192.168.203.118@tcp11 [8/256/0/180] [13359.073189] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13359.089174] LNet: Added LNI 192.168.203.118@tcp12 [8/256/0/180] [13362.033574] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13362.059560] LNet: Added LNI 192.168.203.118@tcp13 [8/256/0/180] [13365.383041] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13365.418125] LNet: Added LNI 192.168.203.118@tcp14 [8/256/0/180] [13368.536529] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13368.558857] LNet: Added LNI 192.168.203.118@tcp15 [8/256/0/180] [13371.593203] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13371.622846] LNet: Added LNI 192.168.203.118@tcp16 [8/256/0/180] [13374.864827] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13374.900461] LNet: Added LNI 192.168.203.118@tcp17 [8/256/0/180] [13378.262200] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13378.299625] LNet: Added LNI 192.168.203.118@tcp18 [8/256/0/180] [13381.420577] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13381.441272] LNet: Added LNI 192.168.203.118@tcp19 [8/256/0/180] [13384.944971] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13384.965519] LNet: Added LNI 192.168.203.118@tcp20 [8/256/0/180] [13388.542750] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13388.571461] LNet: Added LNI 192.168.203.118@tcp21 [8/256/0/180] [13392.480503] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13392.513781] LNet: Added LNI 192.168.203.118@tcp22 [8/256/0/180] [13396.289478] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13396.310946] LNet: Added LNI 192.168.203.118@tcp23 [8/256/0/180] [13400.405651] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13400.433510] LNet: Added LNI 192.168.203.118@tcp24 [8/256/0/180] [13404.697333] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13404.725361] LNet: Added LNI 192.168.203.118@tcp25 [8/256/0/180] [13409.297713] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13409.320631] LNet: Added LNI 192.168.203.118@tcp26 [8/256/0/180] [13414.263749] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13414.292710] LNet: Added LNI 192.168.203.118@tcp27 [8/256/0/180] [13419.005174] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13419.037427] LNet: Added LNI 192.168.203.118@tcp28 [8/256/0/180] [13423.694407] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13423.741060] LNet: Added LNI 192.168.203.118@tcp29 [8/256/0/180] [13427.977496] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13427.999082] LNet: Added LNI 192.168.203.118@tcp30 [8/256/0/180] [13432.034751] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13432.062358] LNet: Added LNI 192.168.203.118@tcp31 [8/256/0/180] [13436.240291] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13436.260744] LNet: Added LNI 192.168.203.118@tcp32 [8/256/0/180] [13440.657832] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13440.680652] LNet: Added LNI 192.168.203.118@tcp33 [8/256/0/180] [13445.025332] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13445.047903] LNet: Added LNI 192.168.203.118@tcp34 [8/256/0/180] [13449.209943] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13449.234281] LNet: Added LNI 192.168.203.118@tcp35 [8/256/0/180] [13453.748593] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13453.768721] LNet: Added LNI 192.168.203.118@tcp36 [8/256/0/180] [13458.549952] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13458.579851] LNet: Added LNI 192.168.203.118@tcp37 [8/256/0/180] [13463.601307] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13463.626916] LNet: Added LNI 192.168.203.118@tcp38 [8/256/0/180] [13469.032106] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13469.062362] LNet: Added LNI 192.168.203.118@tcp39 [8/256/0/180] [13474.570772] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13474.609996] LNet: Added LNI 192.168.203.118@tcp40 [8/256/0/180] [13479.864784] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13479.897655] LNet: Added LNI 192.168.203.118@tcp41 [8/256/0/180] [13485.550056] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13485.568828] LNet: Added LNI 192.168.203.118@tcp42 [8/256/0/180] [13490.753149] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13490.775326] LNet: Added LNI 192.168.203.118@tcp43 [8/256/0/180] [13495.575941] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13495.602506] LNet: Added LNI 192.168.203.118@tcp44 [8/256/0/180] [13500.370750] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13500.388689] LNet: Added LNI 192.168.203.118@tcp45 [8/256/0/180] [13505.690732] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13505.723712] LNet: Added LNI 192.168.203.118@tcp46 [8/256/0/180] [13510.743166] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13510.769569] LNet: Added LNI 192.168.203.118@tcp47 [8/256/0/180] [13515.467576] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13515.491841] LNet: Added LNI 192.168.203.118@tcp48 [8/256/0/180] [13520.097470] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13520.128970] LNet: Added LNI 192.168.203.118@tcp49 [8/256/0/180] [13524.771309] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13524.789890] LNet: Added LNI 192.168.203.118@tcp50 [8/256/0/180] [13538.786138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13538.792754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13538.801588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13539.611457] Lustre: server umount lustre-MDT0000 complete [13540.843092] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13540.853684] Lustre: Skipped 1 previous similar message [13542.159441] LustreError: 443406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344099 with bad export cookie 2270280951967358881 [13542.165389] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13542.168220] LustreError: 443406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13542.361471] Lustre: server umount lustre-MDT0001 complete [13554.959459] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [13560.785435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13562.335464] Lustre: 442685:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780344103/real 1780344103] req@000000000d66e8d7 x1866825838227584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780344119 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13562.360550] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13564.191169] Lustre: 442684:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780344105/real 1780344105] req@00000000e57a47c3 x1866825838227712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780344121 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13566.495418] LustreError: 442683:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000059f33134 x1866825838228736/t0(0) o250->MGC192.168.203.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13566.726703] LustreError: 445146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13566.746406] LustreError: 445146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13566.784776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13567.455336] Lustre: 442687:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780344108/real 1780344108] req@00000000922aece2 x1866825838227840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780344124 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13567.775744] LustreError: 445147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13569.684823] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [13571.879277] LustreError: 445146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13571.901102] LustreError: 445146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13572.495089] Lustre: 442685:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780344114/real 1780344114] req@000000003b0f0916 x1866825838228224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780344130 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13572.514758] Lustre: 442685:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [13575.889573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13576.209284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13577.119219] Lustre: 442685:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780344118/real 1780344118] req@00000000b4ccc105 x1866825838228480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780344134 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13579.304594] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [13581.292641] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13582.309259] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13582.942464] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13585.234270] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13601.766323] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13601.775596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13601.797136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13602.793849] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13604.469745] Lustre: server umount lustre-OST0000 complete [13606.880069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13606.885439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13606.894978] Lustre: Skipped 1 previous similar message [13606.898421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13606.904741] Lustre: Skipped 2 previous similar messages [13612.868219] Lustre: server umount lustre-MDT0000 complete [13613.025296] LustreError: 468523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13613.037803] LustreError: 468523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13615.158168] LustreError: 443405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344172 with bad export cookie 2270280951967361247 [13615.166414] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13615.167624] LustreError: 443405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13615.389779] Lustre: server umount lustre-MDT0001 complete [13621.999912] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [13623.738705] Key type lgssc unregistered [13624.036782] LNet: 471054:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13624.047673] LNetError: 471054:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13624.052332] LNet: Removed LNI 192.168.203.118@tcp [13624.053948] LNet: Removed LNI 192.168.203.118@tcp10 [13624.056172] LNet: Removed LNI 192.168.203.118@tcp11 [13624.058163] LNet: Removed LNI 192.168.203.118@tcp12 [13624.060687] LNet: Removed LNI 192.168.203.118@tcp13 [13624.062954] LNet: Removed LNI 192.168.203.118@tcp14 [13624.065073] LNet: Removed LNI 192.168.203.118@tcp15 [13624.067980] LNet: Removed LNI 192.168.203.118@tcp16 [13624.070989] LNet: Removed LNI 192.168.203.118@tcp17 [13624.073396] LNet: Removed LNI 192.168.203.118@tcp18 [13624.075394] LNet: Removed LNI 192.168.203.118@tcp19 [13624.077552] LNet: Removed LNI 192.168.203.118@tcp20 [13624.079669] LNet: Removed LNI 192.168.203.118@tcp21 [13624.081829] LNet: Removed LNI 192.168.203.118@tcp22 [13624.084132] LNet: Removed LNI 192.168.203.118@tcp23 [13624.086092] LNet: Removed LNI 192.168.203.118@tcp24 [13624.087801] LNet: Removed LNI 192.168.203.118@tcp25 [13624.089368] LNet: Removed LNI 192.168.203.118@tcp26 [13624.091346] LNet: Removed LNI 192.168.203.118@tcp27 [13624.092855] LNet: Removed LNI 192.168.203.118@tcp28 [13624.094680] LNet: Removed LNI 192.168.203.118@tcp29 [13624.096447] LNet: Removed LNI 192.168.203.118@tcp30 [13624.098746] LNet: Removed LNI 192.168.203.118@tcp31 [13624.100818] LNet: Removed LNI 192.168.203.118@tcp32 [13624.102902] LNet: Removed LNI 192.168.203.118@tcp33 [13624.105250] LNet: Removed LNI 192.168.203.118@tcp34 [13624.107268] LNet: Removed LNI 192.168.203.118@tcp35 [13624.109269] LNet: Removed LNI 192.168.203.118@tcp36 [13624.112344] LNet: Removed LNI 192.168.203.118@tcp37 [13624.117586] LNet: Removed LNI 192.168.203.118@tcp38 [13624.120417] LNet: Removed LNI 192.168.203.118@tcp39 [13624.122120] LNet: Removed LNI 192.168.203.118@tcp40 [13624.124439] LNet: Removed LNI 192.168.203.118@tcp41 [13624.127262] LNet: Removed LNI 192.168.203.118@tcp42 [13624.129560] LNet: Removed LNI 192.168.203.118@tcp43 [13624.132551] LNet: Removed LNI 192.168.203.118@tcp44 [13624.134822] LNet: Removed LNI 192.168.203.118@tcp45 [13624.137332] LNet: Removed LNI 192.168.203.118@tcp46 [13624.139911] LNet: Removed LNI 192.168.203.118@tcp47 [13624.142166] LNet: Removed LNI 192.168.203.118@tcp48 [13624.144657] LNet: Removed LNI 192.168.203.118@tcp49 [13624.152898] LNet: Removed LNI 192.168.203.118@tcp50 [13624.584391] Key type .llcrypt unregistered [13624.586970] Key type ._llcrypt unregistered [13637.518428] Key type ._llcrypt registered [13637.523486] Key type .llcrypt registered [13637.633743] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [13648.564330] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [13649.360835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13649.392427] alg: No test for adler32 (adler32-zlib) [13650.392656] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [13650.561900] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [13652.215181] Key type lgssc registered [13652.932827] Lustre: Echo OBD driver; http://www.lustre.org/ [13674.503610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13674.519174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13675.659502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13675.682992] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13675.738443] Lustre: lustre-MDT0000: new disk, initializing [13675.793542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13675.805690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13678.368780] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [13685.436877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13685.520233] Lustre: 474191:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13685.562864] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13685.566870] Lustre: Skipped 1 previous similar message [13685.624143] Lustre: lustre-MDT0001: new disk, initializing [13685.708339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13685.743256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13685.748943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13688.612310] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [13693.787972] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13696.301156] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13701.318985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13701.432909] Lustre: *** cfs_fail_loc=90f, val=0*** [13701.491233] Lustre: lustre-OST0000: new disk, initializing [13701.494511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13701.501679] Lustre: 475980:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13701.572796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13705.875968] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [13709.384251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13709.392092] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13709.489610] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13711.681262] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13720.037718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13720.039346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13720.040558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13720.048925] Lustre: Skipped 1 previous similar message [13720.055163] LustreError: Skipped 1 previous similar message [13722.592223] Lustre: server umount lustre-OST0000 complete [13725.154388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13725.155953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13725.166688] Lustre: Skipped 1 previous similar message [13725.170608] Lustre: Skipped 1 previous similar message [13730.272388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13730.275305] Lustre: Skipped 2 previous similar messages [13731.199337] Lustre: server umount lustre-MDT0000 complete [13733.434300] LustreError: 474183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344290 with bad export cookie 8802254293626056823 [13733.436181] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13733.440694] LustreError: 474183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13733.620148] Lustre: server umount lustre-MDT0001 complete [13739.987750] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [13741.529571] Key type lgssc unregistered [13741.693948] LNet: 477753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13741.698462] LNetError: 477753:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13741.713764] LNet: Removed LNI 192.168.203.118@tcp [13742.178691] Key type .llcrypt unregistered [13742.180567] Key type ._llcrypt unregistered [13754.171295] Key type ._llcrypt registered [13754.172792] Key type .llcrypt registered [13754.247585] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [13754.782604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13754.794749] alg: No test for adler32 (adler32-zlib) [13755.733470] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [13755.919448] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [13757.551144] Key type lgssc registered [13758.272647] Lustre: Echo OBD driver; http://www.lustre.org/ [13762.958921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13762.971686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13764.284659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13766.611612] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [13772.158382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13772.374396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13775.049144] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [13778.144458] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13780.217203] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13784.409853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13784.605631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13788.394707] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [13791.779651] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13803.139926] loop0: detected capacity change from 0 to 204800000 [13803.241613] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13803.247358] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13803.254063] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13806.595541] loop0: detected capacity change from 0 to 204800000 [13806.692214] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13806.697974] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13806.704943] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13809.771242] loop0: detected capacity change from 0 to 204800000 [13812.665193] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13812.746067] Lustre: fs15246-OST0000: new disk, initializing [13812.748931] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [13812.753741] Lustre: 482248:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13812.789498] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13816.023435] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [13821.622762] loop1: detected capacity change from 0 to 204800000 [13824.714496] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13824.797300] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [13824.826467] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [13824.875134] Lustre: fs15246-MDT0000: new disk, initializing [13824.966587] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13824.989365] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13825.049769] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [13826.081028] Lustre: 483351:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000edd162e3 x1866826318385152/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1780344399 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [13826.929151] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [13835.173587] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13835.182460] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13835.201915] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13838.821749] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13838.833153] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [13843.938138] LustreError: 481143:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13843.949954] LustreError: 481143:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13843.962583] Lustre: server umount fs15246-MDT0000 complete [13848.762127] Lustre: server umount fs15246-OST0000 complete [13854.179458] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13854.182519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13854.195443] Lustre: Skipped 1 previous similar message [13859.298688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13859.303159] Lustre: Skipped 2 previous similar messages [13860.005689] Lustre: server umount lustre-OST0000 complete [13864.418191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13864.419710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13864.422387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13864.446169] Lustre: Skipped 1 previous similar message [13869.537277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13869.544480] Lustre: Skipped 2 previous similar messages [13876.191118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13876.282689] Lustre: server umount lustre-MDT0000 complete [13878.070526] LustreError: 478974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344435 with bad export cookie 9725438861659101412 [13878.073956] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13878.076399] LustreError: 478974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13878.225618] Lustre: server umount lustre-MDT0001 complete [13883.205403] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [13884.666395] Key type lgssc unregistered [13884.832724] LNet: 486410:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13884.839398] LNetError: 486410:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13884.859658] LNet: Removed LNI 192.168.203.118@tcp [13885.241735] Key type .llcrypt unregistered [13885.243696] Key type ._llcrypt unregistered [13895.589621] Key type ._llcrypt registered [13895.591443] Key type .llcrypt registered [13895.671137] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [13896.245047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13896.260536] alg: No test for adler32 (adler32-zlib) [13897.175848] Lustre: Lustre: Build Version: 2.17.53_30_g468b778 [13897.307085] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [13898.927253] Key type lgssc registered [13899.467838] Lustre: Echo OBD driver; http://www.lustre.org/ [13919.215325] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [13923.925954] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [13941.274903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13941.284640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13942.395348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13942.412223] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13942.451978] Lustre: lustre-MDT0000: new disk, initializing [13942.494377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13942.508475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13944.443546] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [13950.825150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13950.910986] Lustre: 490800:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13950.967661] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13950.976095] Lustre: Skipped 1 previous similar message [13951.039783] Lustre: lustre-MDT0001: new disk, initializing [13951.114766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13951.150899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13951.160092] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13953.243242] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [13957.568361] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13959.265161] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13962.443159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13962.569692] Lustre: lustre-OST0000: new disk, initializing [13962.572125] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13962.576814] Lustre: 492541:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13962.613399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13963.479060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13963.486680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13963.524361] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13965.372984] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [13969.677826] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13971.723672] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13971.837845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13973.726693] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13973.861551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13976.870488] Lustre: server umount lustre-OST0000 complete [13979.109109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13979.110478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13979.122798] Lustre: Skipped 1 previous similar message [13979.131671] Lustre: Skipped 1 previous similar message [13984.225876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13984.228846] Lustre: Skipped 1 previous similar message [13989.346899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13989.351019] Lustre: Skipped 1 previous similar message [13989.395986] Lustre: server umount lustre-MDT0000 complete [13991.267251] LustreError: 490791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344548 with bad export cookie 9795660829965367320 [13991.272535] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13991.276636] LustreError: 490791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14000.674516] Lustre: server umount lustre-MDT0001 complete [14013.277365] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [14018.371493] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [14033.470260] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13673 sec ========= 16:09:49 (1780344589) [14034.271811] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:09:50 (1780344590) === [14035.953637] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:09:51 (1780344591) === [14054.681529] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [14055.983806] Key type lgssc unregistered [14056.137684] LNet: 498974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14056.145084] LNetError: 498974:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14056.154202] LNet: Removed LNI 192.168.203.118@tcp [14056.551212] Key type .llcrypt unregistered [14056.553259] Key type ._llcrypt unregistered