[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 549987460 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002524] x2apic enabled [ 0.003012] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007683] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010143] LSM: Security Framework initializing [ 0.011051] Yama: becoming mindful. [ 0.012033] SELinux: Initializing. [ 0.014052] *** VALIDATE selinux *** [ 0.022819] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026664] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029110] *** VALIDATE tmpfs *** [ 0.030430] *** VALIDATE proc *** [ 0.032229] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.035313] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037078] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039042] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.043055] debug: unmapping init [mem 0xffffffff97c59000-0xffffffff97c60fff] [ 0.046182] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047723] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048023] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050011] ... generic registers: 4 [ 0.051013] ... value mask: 0000ffffffffffff [ 0.052014] ... max period: 00007fffffffffff [ 0.053012] ... fixed-purpose events: 3 [ 0.054010] ... event mask: 000000070000000f [ 0.055287] rcu: Hierarchical SRCU implementation. [ 0.057548] smp: Bringing up secondary CPUs ... [ 0.058600] x86: Booting SMP configuration: [ 0.059024] .... node #0, CPUs: #1 #2 #3 [ 0.062154] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.065021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137215] node 0 deferred pages initialised in 69ms [ 0.142033] devtmpfs: initialized [ 0.143219] x86/mm: Memory block size: 128MB [ 0.146868] gcov: version magic: 0x41383552 [ 0.150253] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152287] pinctrl core: initialized pinctrl subsystem [ 0.153164] [ 0.153761] ************************************************************* [ 0.154012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155012] ** ** [ 0.156010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157013] ** ** [ 0.158012] ** This means that this kernel is built to expose internal ** [ 0.159013] ** IOMMU data structures, which may compromise security on ** [ 0.160013] ** your system. ** [ 0.161013] ** ** [ 0.162012] ** If you see this message and you are not debugging the ** [ 0.163012] ** kernel, report this immediately to your vendor! ** [ 0.164013] ** ** [ 0.165012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166015] ************************************************************* [ 0.167703] NET: Registered protocol family 16 [ 0.169323] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.175059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.179109] cpuidle: using governor menu [ 0.181799] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.184650] PCI: Using configuration type 1 for base access [ 0.187131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.197237] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.200043] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.204254] cryptd: max_cpu_qlen set to 1000 [ 0.207275] ACPI: Added _OSI(Module Device) [ 0.209009] ACPI: Added _OSI(Processor Device) [ 0.210010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.212012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.217472] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.225324] ACPI: Interpreter enabled [ 0.227062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.228011] ACPI: Using IOAPIC for interrupt routing [ 0.230110] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.234349] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.245090] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.247040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.250018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.253077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.259523] acpiphp: Slot [2] registered [ 0.261153] acpiphp: Slot [5] registered [ 0.262129] acpiphp: Slot [6] registered [ 0.264109] acpiphp: Slot [7] registered [ 0.266107] acpiphp: Slot [8] registered [ 0.267099] acpiphp: Slot [9] registered [ 0.269206] acpiphp: Slot [10] registered [ 0.271082] acpiphp: Slot [3] registered [ 0.272102] acpiphp: Slot [4] registered [ 0.274099] acpiphp: Slot [11] registered [ 0.276127] acpiphp: Slot [12] registered [ 0.277084] acpiphp: Slot [13] registered [ 0.279105] acpiphp: Slot [14] registered [ 0.280110] acpiphp: Slot [15] registered [ 0.282087] acpiphp: Slot [16] registered [ 0.283075] acpiphp: Slot [17] registered [ 0.285080] acpiphp: Slot [18] registered [ 0.286092] acpiphp: Slot [19] registered [ 0.288119] acpiphp: Slot [20] registered [ 0.289155] acpiphp: Slot [21] registered [ 0.291086] acpiphp: Slot [22] registered [ 0.292135] acpiphp: Slot [23] registered [ 0.294100] acpiphp: Slot [24] registered [ 0.296121] acpiphp: Slot [25] registered [ 0.297159] acpiphp: Slot [26] registered [ 0.299098] acpiphp: Slot [27] registered [ 0.301122] acpiphp: Slot [28] registered [ 0.302085] acpiphp: Slot [29] registered [ 0.304079] acpiphp: Slot [30] registered [ 0.305080] acpiphp: Slot [31] registered [ 0.307094] PCI host bridge to bus 0000:00 [ 0.308016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.311019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.313018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.316023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.319024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.322021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.324153] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.326688] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.330241] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.341942] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.347110] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.350014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.351013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.355020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.358617] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.361159] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.364066] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.370030] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.377034] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.392039] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.398846] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.406457] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.428015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.448015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.479016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.503067] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.510017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.515016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.531014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.540970] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.547015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.553014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.567018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.576540] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.582014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.588020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.602015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.616365] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.624000] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.624000] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.638024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.651556] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.658019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.665017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.680023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.690925] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.693449] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.697397] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.700357] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.702202] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.707143] iommu: Default domain type: Passthrough [ 0.708539] SCSI subsystem initialized [ 0.709308] ACPI: bus type USB registered [ 0.710128] usbcore: registered new interface driver usbfs [ 0.711228] usbcore: registered new interface driver hub [ 0.712124] usbcore: registered new device driver usb [ 0.714137] pps_core: LinuxPPS API ver. 1 registered [ 0.716011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.719080] PTP clock support registered [ 0.722090] EDAC MC: Ver: 3.0.0 [ 0.724130] PCI: Using ACPI for IRQ routing [ 0.725890] NetLabel: Initializing [ 0.727009] NetLabel: domain hash size = 128 [ 0.729010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.731090] NetLabel: unlabeled traffic allowed by default [ 0.734055] vgaarb: loaded [ 0.735240] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.737013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.745000] clocksource: Switched to clocksource kvm-clock [ 0.852112] VFS: Disk quotas dquot_6.6.0 [ 0.853677] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.856422] *** VALIDATE ramfs *** [ 0.857602] *** VALIDATE hugetlbfs *** [ 0.859099] pnp: PnP ACPI init [ 0.861390] pnp: PnP ACPI: found 6 devices [ 0.912330] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.916406] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.919224] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.921846] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.924838] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.927662] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.931015] NET: Registered protocol family 2 [ 0.934166] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.939705] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.944790] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.951234] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.955125] TCP: Hash tables configured (established 65536 bind 65536) [ 0.958397] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.961757] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.964933] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.969137] NET: Registered protocol family 1 [ 0.973126] RPC: Registered named UNIX socket transport module. [ 0.975635] RPC: Registered udp transport module. [ 0.977655] RPC: Registered tcp transport module. [ 0.979584] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.982163] NET: Registered protocol family 44 [ 0.984049] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.986537] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.989021] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.991688] PCI: CLS 0 bytes, default 64 [ 0.993656] Unpacking initramfs... [ 2.454675] debug: unmapping init [mem 0xffff9a257cc54000-0xffff9a257ffbffff] [ 2.460892] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.463412] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.467208] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.970362] Initialise system trusted keyrings [ 2.972424] Key type blacklist registered [ 2.974943] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.983854] zbud: loaded [ 2.987794] *** VALIDATE nfs *** [ 2.989375] *** VALIDATE nfs4 *** [ 2.991199] pstore: using deflate compression [ 2.995645] Platform Keyring initialized [ 3.108611] NET: Registered protocol family 38 [ 3.110783] Key type asymmetric registered [ 3.112666] Asymmetric key parser 'x509' registered [ 3.114967] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.118287] io scheduler mq-deadline registered [ 3.120304] io scheduler kyber registered [ 3.122074] io scheduler bfq registered [ 3.125254] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.128722] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.132116] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.135499] ACPI: Power Button [PWRF] [ 3.141372] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.149145] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.177091] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.184564] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.198194] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.229344] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.259143] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.265541] Non-volatile memory driver v1.3 [ 3.267707] Linux agpgart interface v0.103 [ 3.301046] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.304531] vda: detected capacity change from 0 to 76779520 [ 3.321525] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.325052] vdb: detected capacity change from 0 to 1073741824 [ 3.341899] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.346144] vdc: detected capacity change from 0 to 2621440000 [ 3.361383] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.365666] vdd: detected capacity change from 0 to 2621440000 [ 3.382076] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.385301] vde: detected capacity change from 0 to 4294967296 [ 3.407643] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.410822] vdf: detected capacity change from 0 to 4294967296 [ 3.416797] libphy: Fixed MDIO Bus: probed [ 3.422226] usbcore: registered new interface driver usbserial_generic [ 3.424683] usbserial: USB Serial support registered for generic [ 3.427093] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.431603] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.433308] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.436210] mousedev: PS/2 mouse device common for all mice [ 3.438869] rtc_cmos 00:05: RTC can wake from S4 [ 3.441366] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.443283] rtc_cmos 00:05: registered as rtc0 [ 3.448598] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.448659] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.454524] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.456230] intel_pstate: CPU model not supported [ 3.463700] hid: raw HID events driver (C) Jiri Kosina [ 3.466217] usbcore: registered new interface driver usbhid [ 3.468231] usbhid: USB HID core driver [ 3.469898] drop_monitor: Initializing network drop monitor service [ 3.472384] Initializing XFRM netlink socket [ 3.474646] NET: Registered protocol family 10 [ 3.478349] Segment Routing with IPv6 [ 3.480259] NET: Registered protocol family 17 [ 3.483088] mpls_gso: MPLS GSO support [ 3.489654] RAS: Correctable Errors collector initialized. [ 3.491899] AVX version of gcm_enc/dec engaged. [ 3.493766] AES CTR mode by8 optimization enabled [ 3.572278] sched_clock: Marking stable (3572246673, 0)->(4596125408, -1023878735) [ 3.576099] registered taskstats version 1 [ 3.578287] Loading compiled-in X.509 certificates [ 3.581532] zswap: loaded using pool lzo/zbud [ 3.607147] Key type big_key registered [ 3.620393] Key type encrypted registered [ 3.622185] ima: No TPM chip found, activating TPM-bypass! [ 3.624556] ima: Allocated hash algorithm: sha1 [ 3.626502] ima: No architecture policies found [ 3.628464] evm: Initialising EVM extended attributes: [ 3.630571] evm: security.selinux [ 3.631899] evm: security.ima [ 3.633080] evm: security.capability [ 3.634446] evm: HMAC attrs: 0x1 [ 3.637077] rtc_cmos 00:05: setting system clock to 2026-09-08 06:58:42 UTC (1788850722) [ 3.643525] debug: unmapping init [mem 0xffffffff98c03000-0xffffffff98dfffff] [ 3.646517] debug: unmapping init [mem 0xffffffff97982000-0xffffffff97c58fff] [ 3.656074] Write protecting the kernel read-only data: 28672k [ 3.659521] debug: unmapping init [mem 0xffffffff96003000-0xffffffff961fffff] [ 3.662400] debug: unmapping init [mem 0xffffffff96914000-0xffffffff969fffff] [ 3.696934] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.705382] systemd[1]: Detected virtualization kvm. [ 3.707337] systemd[1]: Detected architecture x86-64. [ 3.709204] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.735828] systemd[1]: No hostname configured. [ 3.737503] systemd[1]: Set hostname to . [ 3.739512] random: systemd: uninitialized urandom read (16 bytes read) [ 3.741842] systemd[1]: Initializing machine ID from random generator. [ 3.888789] random: systemd: uninitialized urandom read (16 bytes read) [ 3.891683] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.896416] random: systemd: uninitialized urandom read (16 bytes read) [ 3.899254] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.903650] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ 4.073753] random: fast init done Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.597552] device-mapper: uevent: version 1.0.3 [ 4.601833] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.315813] virtio_net virtio0 ens2: renamed from eth0 [ 5.336397] scsi host0: ata_piix [ 5.360763] scsi host1: ata_piix [ 5.362120] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.365317] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.517276] dracut-initqueue[598]: RTNETLINK answers: File exists [ 10.139647] random: crng init done [ 10.141444] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.449991] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.736585] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.019836] SELinux: Disabled at runtime. [ 12.082986] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.092821] systemd[1]: Detected virtualization kvm. [ 12.094989] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.559896] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.563653] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.568724] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.575359] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.578755] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.587432] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.597367] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Started Journ[ 12.794947] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS al Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.077376] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.334518] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.349924] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.484052] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.493272] EDAC sbridge: Ver: 1.1.2 [ 15.145393] Key type dns_resolver registered [ 15.446120] NFS: Registering the id_resolver key type [ 15.447791] Key type id_resolver registered [ 15.449018] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg316-server login: [ 40.697063] libcfs: loading out-of-tree module taints kernel. [ 40.718957] Key type ._llcrypt registered [ 40.720562] Key type .llcrypt registered [ 40.767212] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 52.202214] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 53.195093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 53.204636] alg: No test for adler32 (adler32-zlib) [ 54.496956] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 55.189736] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 56.879712] Key type lgssc registered [ 58.065647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 69.464354] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 94.937366] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 103.998825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 104.022530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 105.173587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 105.210571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 105.279532] Lustre: lustre-MDT0000: new disk, initializing [ 105.365364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 105.407340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 111.183801] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 123.288093] hrtimer: interrupt took 7085449 ns [ 136.943650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 137.196273] Lustre: 6485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 137.263532] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 137.273134] Lustre: Skipped 1 previous similar message [ 137.491407] Lustre: lustre-MDT0001: new disk, initializing [ 137.638436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 137.723266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 137.740633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 143.888148] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 153.372052] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 164.707675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 165.055930] Lustre: lustre-OST0000: new disk, initializing [ 165.058625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 165.064497] Lustre: 8438:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 165.141994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 166.005962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 166.023786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 166.116090] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 171.359877] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.719321] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 184.902652] Lustre: lustre-OST0001: new disk, initializing [ 184.906742] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 184.923087] Lustre: 9512:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 184.999418] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 191.373561] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.145071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 194.158823] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 194.234415] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 202.934903] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 209.211655] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 215.483525] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing check_logdir /tmp/testlogs/ [ 220.521129] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing yml_node [ 224.775657] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 227.010255] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 229.360979] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 231.068478] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 03:02:28 EDT 2026 [ 246.964689] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 248.201451] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 265.696165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 265.714391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 265.746270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 266.243634] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 266.265904] Lustre: Skipped 2 previous similar messages [ 266.272835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 266.279433] Lustre: Skipped 3 previous similar messages [ 270.455397] Lustre: server umount lustre-MDT0000 complete [ 274.256343] LustreError: 6478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850993 with bad export cookie 15747985883465822549 [ 274.257051] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 274.278630] LustreError: 6478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 276.452890] LustreError: 6491:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 276.455452] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 276.456922] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 276.474300] LustreError: 6491:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 276.506595] Lustre: Skipped 1 previous similar message [ 279.457075] LustreError: 6491:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 279.458138] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 279.474187] Lustre: Skipped 2 previous similar messages [ 280.789362] Lustre: server umount lustre-MDT0001 complete [ 284.314915] Lustre: server umount lustre-OST0000 complete [ 287.957756] Lustre: server umount lustre-OST0001 complete [ 294.081911] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 302.728584] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 338.862050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 339.082331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 339.132943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 339.218047] Lustre: lustre-MDT0000: new disk, initializing [ 339.310692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 339.339776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 344.010808] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 355.340603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 355.505929] Lustre: 15061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 355.586102] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 355.595773] Lustre: Skipped 1 previous similar message [ 355.725265] Lustre: lustre-MDT0001: new disk, initializing [ 355.851454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 355.895670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 355.913875] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 360.524882] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 371.579837] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 378.267905] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 386.384791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 386.874926] Lustre: lustre-OST0000: new disk, initializing [ 386.885788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 386.908112] Lustre: 17030:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 387.079707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 388.474130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 388.489547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 388.678582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 394.228597] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 404.679417] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 409.783701] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 410.010191] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 414.783954] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 414.961891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 419.299492] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 419.306689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 419.332806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 419.338934] Lustre: Skipped 1 previous similar message [ 429.568322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 429.581758] Lustre: Skipped 3 previous similar messages [ 431.071149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 431.371600] Lustre: server umount lustre-OST0000 complete [ 437.728483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 437.740251] LustreError: Skipped 1 previous similar message [ 437.756086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 437.781067] Lustre: Skipped 1 previous similar message [ 442.105162] Lustre: server umount lustre-MDT0000 complete [ 444.896333] LustreError: 15944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 444.916907] LustreError: 15944:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 446.254540] LustreError: 15054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851165 with bad export cookie 15747985883465824075 [ 446.265589] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 446.813201] Lustre: server umount lustre-MDT0001 complete [ 451.058345] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 03:06:08 (1788851168) [ 453.651587] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 455.646822] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 03:06:12 (1788851172) [ 463.192176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 463.682212] LustreError: 19445:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 463.763878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 468.494239] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 468.963174] LustreError: 19446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 476.952268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.415539] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 488.556531] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 494.549115] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 500.615907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 506.931327] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 514.896482] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 523.748851] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 523.760374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 523.790811] Lustre: Skipped 1 previous similar message [ 523.794906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 523.801525] Lustre: Skipped 3 previous similar messages [ 528.770584] Lustre: server umount lustre-OST0000 complete [ 533.472701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 533.491950] LustreError: Skipped 1 previous similar message [ 547.295441] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 547.464863] Lustre: server umount lustre-MDT0000 complete [ 549.344721] LustreError: 19446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.365818] LustreError: 19446:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 551.075849] LustreError: 19426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851269 with bad export cookie 15747985883465824957 [ 551.081222] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 551.087986] LustreError: 19426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 551.530816] Lustre: server umount lustre-MDT0001 complete [ 562.076558] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 564.875778] Key type lgssc unregistered [ 565.234133] LNet: 22975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 565.246982] LNetError: 22975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 565.265340] LNet: Removed LNI 192.168.203.116@tcp [ 566.208150] Key type .llcrypt unregistered [ 566.211495] Key type ._llcrypt unregistered [ 588.331814] Key type ._llcrypt registered [ 588.334676] Key type .llcrypt registered [ 588.437740] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 589.168834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 589.299190] alg: No test for adler32 (adler32-zlib) [ 590.368180] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 590.783789] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 592.521703] Key type lgssc registered [ 593.951454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 601.646333] loop: module loaded [ 601.721602] loop0: detected capacity change from 0 to 204800000 [ 601.956431] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 601.973689] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 602.355670] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 607.886132] loop0: detected capacity change from 0 to 204800000 [ 607.993856] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 608.003644] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 608.016118] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 612.907783] loop0: detected capacity change from 0 to 204800000 [ 618.623212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 618.649187] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 619.859357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 619.944790] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 620.077973] Lustre: lustre-MDT0000: new disk, initializing [ 620.216916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 620.250910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 626.469902] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 635.788963] loop1: detected capacity change from 0 to 204800000 [ 641.105837] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 641.439513] Lustre: lustre-OST07c6: new disk, initializing [ 641.445397] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 641.447908] Lustre: Skipped 1 previous similar message [ 641.453777] Lustre: 26155:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 641.548085] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 647.203855] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 647.279640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 647.291739] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 647.357274] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 652.222744] Lustre: 25142:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 652.232786] Lustre: 25142:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 652.239696] Lustre: 25142:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 652.245059] Lustre: 25142:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 652.250938] Lustre: 25142:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 652.256346] Lustre: 25142:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 666.487678] Lustre: 25142:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 666.500081] Lustre: 25142:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 666.506547] Lustre: 25142:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 666.518689] Lustre: 25142:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 666.527826] Lustre: 25142:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 666.534221] Lustre: 25142:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 671.031215] Lustre: 26673:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 671.043974] Lustre: 26673:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 671.059548] Lustre: 26673:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 671.067845] Lustre: 26673:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 671.080981] Lustre: 26673:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 671.098760] Lustre: 26673:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 671.110687] Lustre: 26673:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 671.116459] Lustre: 26673:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 671.123783] Lustre: 26673:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 671.131493] Lustre: 26673:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 671.136550] Lustre: 26673:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 671.141318] Lustre: 26673:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 674.305236] Lustre: 25143:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 674.316375] Lustre: 25143:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 674.327668] Lustre: 25143:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 674.342132] Lustre: 25143:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 674.348660] Lustre: 25143:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 674.360155] Lustre: 25143:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 674.381671] Lustre: 25143:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 674.390911] Lustre: 25143:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 674.403367] Lustre: 25143:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 674.411363] Lustre: 25143:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 674.420870] Lustre: 25143:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 674.432090] Lustre: 25143:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 679.112351] Lustre: 25144:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 679.118995] Lustre: 25144:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 679.133920] Lustre: 25144:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 679.146720] Lustre: 25144:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 679.152918] Lustre: 25144:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 679.159621] Lustre: 25144:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 679.164957] Lustre: 25144:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 679.171103] Lustre: 25144:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 679.176019] Lustre: 25144:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 679.182020] Lustre: 25144:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 679.191897] Lustre: 25144:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 679.196608] Lustre: 25144:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 688.732397] Lustre: 26673:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 688.750689] Lustre: 26673:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 688.759354] Lustre: 26673:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 688.767481] Lustre: 26673:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 688.777715] Lustre: 26673:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 688.783234] Lustre: 26673:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 688.789223] Lustre: 26673:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 688.794845] Lustre: 26673:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 688.800324] Lustre: 26673:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 688.807309] Lustre: 26673:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 688.816431] Lustre: 26673:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 688.825585] Lustre: 26673:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 739.296230] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 739.308527] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 739.323922] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 743.314541] Lustre: server umount lustre-OST07c6 complete [ 753.250304] Lustre: server umount lustre-MDT0000 complete [ 771.286792] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 774.017264] Key type lgssc unregistered [ 774.270815] LNet: 28720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 774.286493] LNetError: 28720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 774.317296] LNet: Removed LNI 192.168.203.116@tcp [ 775.139122] Key type .llcrypt unregistered [ 775.141384] Key type ._llcrypt unregistered [ 791.679806] Key type ._llcrypt registered [ 791.681704] Key type .llcrypt registered [ 791.768274] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 792.433229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 792.458706] alg: No test for adler32 (adler32-zlib) [ 793.514491] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 793.736867] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 795.383350] Key type lgssc registered [ 796.363166] Lustre: Echo OBD driver; http://www.lustre.org/ [ 803.384815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 803.420341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 804.830309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 808.682955] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 815.075726] LustreError: 29986:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 815.093471] LustreError: 29986:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 817.017684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 817.277827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 822.304264] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 831.500764] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 837.730504] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 845.446717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 846.145852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 849.238279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 855.304153] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 866.576939] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 883.170513] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 883.171406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 883.198199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 883.215755] Lustre: Skipped 1 previous similar message [ 888.291739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 888.298059] Lustre: Skipped 1 previous similar message [ 888.901017] Lustre: server umount lustre-OST0000 complete [ 893.414650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 893.418411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 893.424502] Lustre: Skipped 1 previous similar message [ 893.433659] Lustre: Skipped 2 previous similar messages [ 898.531356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 898.539227] Lustre: Skipped 1 previous similar message [ 898.997500] Lustre: server umount lustre-MDT0000 complete [ 902.616204] LustreError: 29968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851621 with bad export cookie 16212251611463804538 [ 902.619206] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 902.628562] LustreError: 29968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 903.028260] Lustre: server umount lustre-MDT0001 complete [ 913.828765] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 916.334646] Key type lgssc unregistered [ 916.693154] LNet: 33725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.705126] LNetError: 33725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.734250] LNet: Removed LNI 192.168.203.116@tcp [ 918.688385] Key type .llcrypt unregistered [ 918.690498] Key type ._llcrypt unregistered [ 939.553089] Key type ._llcrypt registered [ 939.557400] Key type .llcrypt registered [ 939.733310] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 941.080344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 941.177335] alg: No test for adler32 (adler32-zlib) [ 942.184336] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 942.430038] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 944.087251] Key type lgssc registered [ 945.045217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 952.881969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 952.905491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 954.233866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 958.453435] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 964.580663] LustreError: 34990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 964.606403] LustreError: 34990:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 965.776112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 966.133691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 970.066706] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 976.527316] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 981.745693] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 987.778252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 988.058592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.652622] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 999.356878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 999.479895] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1009.636334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1009.638488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1009.641162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1009.665168] Lustre: Skipped 1 previous similar message [ 1013.537800] Lustre: server umount lustre-OST0000 complete [ 1017.312629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1017.323771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1017.335347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1017.346334] Lustre: Skipped 1 previous similar message [ 1019.872061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1019.875789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.548220] Lustre: server umount lustre-MDT0000 complete [ 1024.993380] LustreError: 34987:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1025.004716] LustreError: 34987:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1027.509633] LustreError: 37481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851746 with bad export cookie 3940633263472216890 [ 1027.513777] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1027.526138] LustreError: 37481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1027.817490] Lustre: server umount lustre-MDT0001 complete [ 1037.566509] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1040.222457] Key type lgssc unregistered [ 1040.538133] LNet: 38520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1040.544154] LNetError: 38520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1040.558554] LNet: Removed LNI 192.168.203.116@tcp [ 1041.466633] Key type .llcrypt unregistered [ 1041.469649] Key type ._llcrypt unregistered [ 1062.046933] Key type ._llcrypt registered [ 1062.052030] Key type .llcrypt registered [ 1062.171335] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1063.152413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1063.293469] alg: No test for adler32 (adler32-zlib) [ 1064.362468] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 1064.596777] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1066.319309] Key type lgssc registered [ 1067.535413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1075.159139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1075.186896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1076.703725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1081.018808] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1086.953289] LustreError: 39911:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1086.965193] LustreError: 39911:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1090.745076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1091.149830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1095.978683] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1102.919613] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1108.168700] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1114.788777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1115.040343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1121.062622] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1121.220826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1128.973046] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1135.452494] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 1136.359551] Lustre: server umount lustre-MDT0000 complete [ 1136.611775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1136.620849] Lustre: Skipped 2 previous similar messages [ 1136.621441] LustreError: 39906:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1136.639771] LustreError: 39906:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1139.100027] LustreError: 39894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851857 with bad export cookie 9410166605329042354 [ 1139.104137] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1139.116374] LustreError: 39894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1140.553076] LustreError: 39907:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1140.582070] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (stopping) [ 1141.733692] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.747047] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1144.738741] LustreError: 39908:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1144.739523] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1144.752174] LustreError: 39908:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1145.357701] Lustre: server umount lustre-MDT0001 complete [ 1165.734019] Lustre: server umount lustre-OST0000 complete [ 1182.269164] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1184.473974] Key type lgssc unregistered [ 1184.733223] LNet: 43770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1184.739554] LNetError: 43770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1184.753200] LNet: Removed LNI 192.168.203.116@tcp [ 1185.335198] Key type .llcrypt unregistered [ 1185.337433] Key type ._llcrypt unregistered [ 1206.067380] Key type ._llcrypt registered [ 1206.073206] Key type .llcrypt registered [ 1206.205790] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1207.444239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1207.499270] alg: No test for adler32 (adler32-zlib) [ 1208.593497] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 1208.812418] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1210.567323] Key type lgssc registered [ 1212.191607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1221.673304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1221.704947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1223.490105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1228.697617] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1233.892250] LustreError: 45161:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1233.923864] LustreError: 45161:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1238.562090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1239.203359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1243.744277] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1249.944810] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1255.674682] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1262.728965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1263.137716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1268.301127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1271.068655] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1283.067688] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1291.129768] Lustre: lustre-OST0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 1293.793938] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1293.799481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1293.807673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1293.834246] Lustre: Skipped 1 previous similar message [ 1293.879965] Lustre: Skipped 1 previous similar message [ 1296.195243] Lustre: lustre-OST0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 1296.417619] Lustre: server umount lustre-OST0000 complete [ 1314.018531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1314.028379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1314.036066] Lustre: Skipped 1 previous similar message [ 1319.393315] LustreError: 45162:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1319.406471] Lustre: server umount lustre-MDT0000 complete [ 1322.539349] LustreError: 45143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852041 with bad export cookie 12314286849013407986 [ 1322.551240] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1328.966683] Lustre: server umount lustre-MDT0001 complete [ 1339.871486] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1342.569485] Key type lgssc unregistered [ 1342.880649] LNet: 48873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1342.897418] LNetError: 48873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1342.922876] LNet: Removed LNI 192.168.203.116@tcp [ 1343.764749] Key type .llcrypt unregistered [ 1343.766817] Key type ._llcrypt unregistered [ 1369.236274] Key type ._llcrypt registered [ 1369.238202] Key type .llcrypt registered [ 1369.373461] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1370.276806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1370.366281] alg: No test for adler32 (adler32-zlib) [ 1371.551684] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 1371.700077] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1373.367194] Key type lgssc registered [ 1374.475647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1381.869863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1381.884703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1383.295306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1387.563529] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1393.635638] LustreError: 50288:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1393.655709] LustreError: 50288:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1394.656748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1394.990485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1398.918467] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1405.441794] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1410.313670] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1415.799511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1416.095409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1420.990492] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1427.470843] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1428.411066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1433.721633] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1437.061486] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1449.186333] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-08 3h22m45s [ 1450.510684] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1454.056645] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1454.057944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1454.072932] Lustre: Skipped 2 previous similar messages [ 1454.082574] Lustre: Skipped 2 previous similar messages [ 1455.137742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1455.147307] Lustre: Skipped 2 previous similar messages [ 1457.028842] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 1457.036635] Lustre: Skipped 1 previous similar message [ 1458.895646] Lustre: server umount lustre-MDT0000 complete [ 1461.219940] LustreError: 50282:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1461.253776] LustreError: 50282:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1462.602358] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1467.369260] LustreError: 50282:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1467.400442] LustreError: 50282:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1470.483519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1470.578204] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1470.826501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1470.910718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1471.863177] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1476.265447] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1479.599719] Lustre: DEBUG MARKER: Wait for df (36632) ... [ 1481.118504] Lustre: DEBUG MARKER: done [ 1485.804481] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1485.835046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1485.872912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1486.305144] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1491.847648] Lustre: server umount lustre-OST0000 complete [ 1497.570232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1497.571786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1497.579419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1497.609178] Lustre: Skipped 3 previous similar messages [ 1497.610757] Lustre: Skipped 2 previous similar messages [ 1501.767885] Lustre: server umount lustre-MDT0000 complete [ 1504.737749] LustreError: 51346:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1504.753312] LustreError: 51346:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1505.120761] LustreError: 50267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852223 with bad export cookie 1327232763983842858 [ 1505.130662] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1505.510246] Lustre: server umount lustre-MDT0001 complete [ 1515.647263] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1518.298964] Key type lgssc unregistered [ 1518.605990] LNet: 55205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1518.611070] LNetError: 55205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1518.625955] LNet: Removed LNI 192.168.203.116@tcp [ 1519.662756] Key type .llcrypt unregistered [ 1519.665251] Key type ._llcrypt unregistered [ 1564.466974] Key type ._llcrypt registered [ 1564.475817] Key type .llcrypt registered [ 1564.626891] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1566.217596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1566.401879] alg: No test for adler32 (adler32-zlib) [ 1567.464673] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 1567.922635] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1569.735492] Key type lgssc registered [ 1571.015724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1580.790495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1580.837334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1582.270542] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1582.307514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1582.666119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1588.957927] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1598.015391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1598.074297] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1598.094447] Lustre: 57412:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1598.259051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1603.030805] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1611.233839] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1616.750860] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1622.389863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1622.567127] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1622.827771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1629.435326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1630.918099] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1640.498643] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1647.671816] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-08 3h26m04s [ 1649.640543] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1656.591266] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1658.967450] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1659.053296] LustreError: 59290:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1679.214967] Lustre: lustre-MDT0000: Client 99edecdc-01f1-4155-b044-97c10a4fcc32 (at 192.168.203.16@tcp) reconnecting [ 1699.072203] LustreError: 59290:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1701.370444] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1703.888491] Lustre: DEBUG MARKER: done [ 1711.589931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1711.591319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1711.600070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1711.615250] Lustre: Skipped 1 previous similar message [ 1711.627755] Lustre: Skipped 1 previous similar message [ 1715.421699] Lustre: server umount lustre-OST0000 complete [ 1721.314218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1721.330244] LustreError: Skipped 1 previous similar message [ 1721.338303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1721.367083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1722.440934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1722.447212] Lustre: Skipped 2 previous similar messages [ 1725.521322] Lustre: server umount lustre-MDT0000 complete [ 1729.324410] LustreError: 57405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852448 with bad export cookie 1065702311787238360 [ 1729.334091] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1729.347675] LustreError: 57405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1729.669693] Lustre: server umount lustre-MDT0001 complete [ 1740.875991] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1743.405897] Key type lgssc unregistered [ 1743.679578] LNet: 61749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1743.686096] LNetError: 61749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1743.697169] LNet: Removed LNI 192.168.203.116@tcp [ 1744.371802] Key type .llcrypt unregistered [ 1744.379346] Key type ._llcrypt unregistered [ 1786.767559] Key type ._llcrypt registered [ 1786.769885] Key type .llcrypt registered [ 1786.860842] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1787.702438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1787.891429] alg: No test for adler32 (adler32-zlib) [ 1788.992773] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 1789.202550] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1790.887382] Key type lgssc registered [ 1792.388720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1796.535628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1796.562033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1797.839828] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1797.857980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1797.970277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1798.861380] Lustre: Failing over lustre-MDT0000 [ 1799.252559] Lustre: server umount lustre-MDT0000 complete [ 1805.784796] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 03:28:43 (1788852523) [ 1813.080161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1813.694433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1819.059319] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1833.203130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1833.403401] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1833.491900] Lustre: 64545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1834.022470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1842.500321] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1855.102346] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1861.237820] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1868.640874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1868.856514] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1869.129888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1876.336431] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1878.242796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1885.650409] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1891.510479] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1897.643869] Lustre: server umount lustre-MDT0000 complete [ 1898.998889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1899.016622] Lustre: Skipped 2 previous similar messages [ 1900.833387] LustreError: 64535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852619 with bad export cookie 376220898431761165 [ 1900.840151] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1900.841463] LustreError: 64535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1901.030504] Lustre: server umount lustre-MDT0001 complete [ 1903.726897] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1907.243083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1914.750467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1919.458320] Lustre: 63466:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788852622/real 1788852622] req@0000000094dd20e8 x1875747807469184/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788852638 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1919.477262] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1920.479163] Lustre: 63465:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788852623/real 1788852623] req@00000000ab473581 x1875747807469312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788852639 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1925.599506] Lustre: 63468:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788852628/real 1788852628] req@0000000038ba16b0 x1875747807469440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788852644 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1925.884438] LustreError: 66394:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1925.911622] LustreError: 66394:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1925.969410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1926.049549] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1930.100615] LustreError: 68315:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1930.131821] LustreError: 68315:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1930.759611] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1932.063513] Lustre: 63466:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788852634/real 1788852634] req@00000000818e2eef x1875747807469824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788852650 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1932.097474] Lustre: 63466:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1935.332845] LustreError: 68314:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1935.363262] LustreError: 68314:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1938.415882] LustreError: 68798:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1938.420886] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1938.452447] LustreError: 68798:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1940.566929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1940.940948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1946.116320] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1946.406237] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1954.954326] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1960.592619] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1971.682495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.690790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1971.707034] Lustre: Skipped 2 previous similar messages [ 1971.729469] Lustre: Skipped 2 previous similar messages [ 1975.391996] Lustre: server umount lustre-MDT0000 complete [ 1976.802236] LustreError: 68313:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1976.834024] LustreError: 68313:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1980.492746] LustreError: 64535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852699 with bad export cookie 376220898431763440 [ 1980.497253] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1980.508757] LustreError: 64535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1980.951154] Lustre: server umount lustre-MDT0001 complete [ 1988.025548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1997.288417] Lustre: 63466:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788852700/real 1788852700] req@000000005789bfe7 x1875747807497856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788852716 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1997.312339] Lustre: 63466:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1997.319928] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1997.494868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2005.535932] LustreError: 63464:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cfd18b08 x1875747807498880/t0(0) o250->MGC192.168.203.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2005.855276] LustreError: 69938:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2005.939755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2005.977894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2006.943174] Lustre: 63465:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788852709/real 1788852709] req@00000000dc862440 x1875747807498368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788852725 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2006.987563] Lustre: 63465:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2009.320723] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2016.886693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2022.748736] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2022.910367] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2027.049545] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2031.504844] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2038.237862] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2045.921265] Lustre: server umount lustre-MDT0000 complete [ 2047.457150] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.470682] LustreError: 66395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2047.483612] Lustre: Skipped 1 previous similar message [ 2047.532098] LustreError: 66395:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 2049.399040] LustreError: 64535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852768 with bad export cookie 376220898431764819 [ 2049.420257] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2049.709184] Lustre: server umount lustre-MDT0001 complete [ 2055.526216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2058.491202] Lustre: DEBUG MARKER: files compared the same [ 2069.652241] Lustre: server umount lustre-OST0000 complete [ 2083.839109] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2086.317139] Key type lgssc unregistered [ 2086.614324] LNet: 74273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2086.622352] LNetError: 74273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2087.655442] LNet: Removed LNI 192.168.203.116@tcp [ 2088.473625] Key type .llcrypt unregistered [ 2088.475484] Key type ._llcrypt unregistered [ 2107.704851] Key type ._llcrypt registered [ 2107.706986] Key type .llcrypt registered [ 2107.804489] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2108.499831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2108.584783] alg: No test for adler32 (adler32-zlib) [ 2109.623418] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 2109.840684] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2111.536728] Key type lgssc registered [ 2112.493269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2120.057529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2120.086264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2121.635518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2126.811738] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug +malloc all [ 2131.937088] LustreError: 75666:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2131.971501] LustreError: 75666:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2135.804136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.211479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2140.837583] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug +malloc all [ 2147.680864] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2152.677354] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2158.270498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2158.481751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2164.136081] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug +malloc all [ 2166.709629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2170.758510] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2182.113313] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2182.113463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.147232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2182.170151] Lustre: Skipped 1 previous similar message [ 2186.597378] Lustre: server umount lustre-OST0000 complete [ 2192.352266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2192.352338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2192.360565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2192.360573] Lustre: Skipped 1 previous similar message [ 2192.382172] LustreError: Skipped 1 previous similar message [ 2197.471814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2197.479628] Lustre: Skipped 2 previous similar messages [ 2198.599983] Lustre: server umount lustre-MDT0000 complete [ 2202.591979] LustreError: 75666:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2202.657361] LustreError: 75666:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2203.491705] LustreError: 75647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852922 with bad export cookie 5004707435667357597 [ 2203.513652] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2203.918824] Lustre: server umount lustre-MDT0001 complete [ 2218.491257] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2221.082853] Key type lgssc unregistered [ 2221.379677] LNet: 79196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2221.384487] LNetError: 79196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2221.399801] LNet: Removed LNI 192.168.203.116@tcp [ 2222.066325] Key type .llcrypt unregistered [ 2222.068637] Key type ._llcrypt unregistered [ 2245.041084] Key type ._llcrypt registered [ 2245.042866] Key type .llcrypt registered [ 2245.186297] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2246.193314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2246.242783] alg: No test for adler32 (adler32-zlib) [ 2247.264723] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 2247.419585] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2249.111426] Key type lgssc registered [ 2250.255395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2257.991399] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2258.026359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2273.631646] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2278.752765] LustreError: 80584:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.116@tcp: failed processing log, type 4: rc = -110 [ 2305.439461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2312.907431] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2320.088471] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2326.417417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.567700] Lustre: *** cfs_fail_loc=706, val=0*** [ 2327.097224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2327.176188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2331.923768] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2332.134766] LustreError: 80609:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.181153] LustreError: 80609:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2337.257645] LustreError: 80610:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2339.177689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2339.422370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2342.673075] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2349.444182] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2354.278472] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2357.729180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2357.743500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2357.767446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2359.777655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2359.786178] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2359.820534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2361.313589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2362.745481] Lustre: server umount lustre-OST0000 complete [ 2366.447107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.464722] Lustre: Skipped 1 previous similar message [ 2366.465506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2366.484719] Lustre: Skipped 2 previous similar messages [ 2371.554115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2371.563418] Lustre: Skipped 1 previous similar message [ 2372.636637] Lustre: server umount lustre-MDT0000 complete [ 2376.674246] LustreError: 81674:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2376.686555] LustreError: 81674:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2376.774700] LustreError: 80592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853095 with bad export cookie 6450620853683703695 [ 2376.789598] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2377.187218] Lustre: server umount lustre-MDT0001 complete [ 2385.345940] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2387.961440] Key type lgssc unregistered [ 2388.207640] LNet: 84197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2388.213505] LNetError: 84197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2388.228832] LNet: Removed LNI 192.168.203.116@tcp [ 2388.964954] Key type .llcrypt unregistered [ 2388.967679] Key type ._llcrypt unregistered [ 2412.243086] Key type ._llcrypt registered [ 2412.245466] Key type .llcrypt registered [ 2412.326978] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2413.273588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2413.313496] alg: No test for adler32 (adler32-zlib) [ 2414.336957] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 2414.580411] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2416.215211] Key type lgssc registered [ 2417.295180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2426.640783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2426.658702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2434.638208] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2442.580554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2443.175435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2447.177623] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2448.360753] LustreError: 86299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2448.378416] LustreError: 86299:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2452.450477] LustreError: 86302:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.461228] LustreError: 86302:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2455.245822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2455.687106] LustreError: 86303:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2455.703226] LustreError: 86303:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2455.713665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2461.157285] LustreError: 86299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2461.175137] LustreError: 86299:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2461.236749] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2466.280417] LustreError: 86298:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2466.302501] LustreError: 86298:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2469.790161] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2469.805064] Lustre: MGS: Client db60be9e-1aae-4477-8afd-08d7bd8297c1 (at 0@lo) reconnecting [ 2469.844425] LustreError: 86313:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2470.092379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2470.204901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2474.533623] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2490.860562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2490.863758] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2490.875686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2490.887286] Lustre: Skipped 1 previous similar message [ 2493.694440] Lustre: server umount lustre-OST0000 complete [ 2501.088483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2501.104221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2501.125496] Lustre: Skipped 2 previous similar messages [ 2506.208156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2506.216812] Lustre: Skipped 1 previous similar message [ 2510.319621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2510.322177] Lustre: Skipped 1 previous similar message [ 2512.351195] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2512.420222] Lustre: server umount lustre-MDT0000 complete [ 2515.429060] LustreError: 86302:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2515.449421] LustreError: 86302:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2517.946781] Lustre: server umount lustre-MDT0001 complete [ 2522.376966] Lustre: server umount MGS complete [ 2533.048736] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 03:40:50 (1788853250) [ 2566.214994] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 2577.652672] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2616.963507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2621.643505] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2630.972347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.217522] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2631.268839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2631.339831] Lustre: lustre-MDT0001: new disk, initializing [ 2631.416459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2631.452579] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2631.462316] Lustre: 93472:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2631.476138] LustreError: 93472:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2636.134142] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2646.638209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2646.991336] Lustre: lustre-OST0000: new disk, initializing [ 2646.997115] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2647.008940] Lustre: 94238:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2647.109169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2652.714378] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2656.800784] Lustre: 94714:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000014e16524 x1875748462829952/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788853391 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2661.249583] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2667.614764] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2667.639721] Lustre: MGS: Client 51fc8787-5ce4-44eb-8140-2bcf17b11f6b (at 0@lo) reconnecting [ 2667.665308] LustreError: 93458:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2667.718905] Lustre: 92740:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2667.753942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2667.810813] Lustre: lustre-MDT0000: new disk, initializing [ 2667.948100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2667.999210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2668.195893] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2669.091640] Lustre: 95306:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000008ed0662c x1875748462836352/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788853403 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2673.672109] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2677.232229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2677.239580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2678.187640] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2678.327248] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2704.352969] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2704.366153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2704.389560] Lustre: Skipped 1 previous similar message [ 2704.398552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2707.430289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2712.544506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2712.562153] Lustre: Skipped 3 previous similar messages [ 2718.175470] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2718.343535] Lustre: server umount lustre-OST0000 complete [ 2727.919700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2727.947726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2729.337046] Lustre: server umount lustre-MDT0000 complete [ 2733.024346] LustreError: 93442:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2733.071270] LustreError: 93442:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2741.185456] Lustre: server umount lustre-MDT0001 complete [ 2745.984047] Lustre: server umount MGS complete [ 2759.083707] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 03:44:36 (1788853476) [ 2776.419526] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2780.056460] Key type lgssc unregistered [ 2780.447588] LNet: 97803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2780.454460] LNetError: 97803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2780.473334] LNet: Removed LNI 192.168.203.116@tcp [ 2781.676138] Key type .llcrypt unregistered [ 2781.680389] Key type ._llcrypt unregistered [ 2802.259545] Key type ._llcrypt registered [ 2802.262753] Key type .llcrypt registered [ 2802.386846] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2803.221800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2803.578393] alg: No test for adler32 (adler32-zlib) [ 2804.643624] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 2804.885362] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2806.561149] Key type lgssc registered [ 2807.663273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2814.189655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2814.211969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2815.863525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2820.256845] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2821.092385] LustreError: 99024:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.122714] LustreError: 99024:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2825.190463] LustreError: 99025:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2830.304225] LustreError: 99024:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2830.362720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2830.740482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2835.582273] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2842.261425] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2848.310636] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2855.429421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2855.915278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2861.046539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2863.137415] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2871.006988] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2885.612133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2885.615446] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2885.616306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2885.621474] Lustre: Skipped 1 previous similar message [ 2888.908134] Lustre: server umount lustre-OST0000 complete [ 2892.260941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2892.272369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2892.286994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2892.291638] Lustre: Skipped 1 previous similar message [ 2895.840643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2895.852335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2898.556039] Lustre: server umount lustre-MDT0000 complete [ 2900.961295] LustreError: 99025:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2900.982478] LustreError: 99025:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2903.438350] LustreError: 99005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853622 with bad export cookie 12900743022644405956 [ 2903.439858] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2903.449338] LustreError: 99005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2903.750458] Lustre: server umount lustre-MDT0001 complete [ 2917.160114] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 2920.617880] Key type lgssc unregistered [ 2920.984432] LNet: 102559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2920.989402] LNetError: 102559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2921.002770] LNet: Removed LNI 192.168.203.116@tcp [ 2921.984046] Key type .llcrypt unregistered [ 2921.987965] Key type ._llcrypt unregistered [ 2940.342419] Key type ._llcrypt registered [ 2940.344450] Key type .llcrypt registered [ 2940.474466] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2942.179658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2942.267212] alg: No test for adler32 (adler32-zlib) [ 2944.168688] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 2944.464263] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 2946.151192] Key type lgssc registered [ 2947.261502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2953.178166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2953.187954] LustreError: 103636:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2953.211462] LustreError: 103640:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2953.215859] LustreError: 103636:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4982 [ 2953.223259] LustreError: 103636:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2953.231960] LustreError: 103636:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2953.237248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2954.770983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2962.290896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2962.523468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2967.298388] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2972.729102] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing lsmod [ 2975.169803] LustreError: 104844:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2975.182043] LustreError: 104845:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2975.187795] LustreError: 104844:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 2975.192284] LustreError: 104844:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2975.203331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2975.205358] LustreError: 104844:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2975.643124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2980.834170] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2980.837111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2980.850070] Lustre: Skipped 2 previous similar messages [ 2980.867417] Lustre: Skipped 2 previous similar messages [ 2984.885984] Lustre: server umount lustre-MDT0000 complete [ 2985.959894] LustreError: 103672:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2985.980163] LustreError: 103672:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2988.013257] LustreError: 103675:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2988.359669] LustreError: 103650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853707 with bad export cookie 15777314414574006844 [ 2988.362325] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2988.375026] LustreError: 103650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2990.111233] LustreError: 104106:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000075d72229 x1875749018554368/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2990.356652] Lustre: server umount lustre-MDT0001 complete [ 3004.966983] Lustre: server umount lustre-OST0000 complete [ 3012.911825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3013.406731] LustreError: 105910:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3013.436523] LustreError: 105910:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3013.553084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3018.731949] LustreError: 105911:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3019.761881] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3023.856713] LustreError: 105910:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3031.020623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.607771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3037.198927] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3047.571374] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3055.348693] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3064.320650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3065.178023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3071.530238] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3079.899626] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3083.644461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3093.992623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3094.007872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3094.069656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3094.086680] Lustre: Skipped 1 previous similar message [ 3099.141610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3099.152577] Lustre: Skipped 1 previous similar message [ 3099.957746] Lustre: server umount lustre-OST0000 complete [ 3108.320640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3108.327060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3108.365778] Lustre: Skipped 1 previous similar message [ 3108.373946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3112.389174] Lustre: server umount lustre-MDT0000 complete [ 3114.467799] LustreError: 105905:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3114.493126] LustreError: 105905:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3116.790543] LustreError: 105890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853835 with bad export cookie 15777314414574007558 [ 3116.790827] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3116.804813] LustreError: 105890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3117.080053] Lustre: server umount lustre-MDT0001 complete [ 3127.057739] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3129.610956] Key type lgssc unregistered [ 3129.922718] LNet: 109446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3129.933349] LNetError: 109446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3129.946690] LNet: Removed LNI 192.168.203.116@tcp [ 3130.725835] Key type .llcrypt unregistered [ 3130.728057] Key type ._llcrypt unregistered [ 3154.166787] Key type ._llcrypt registered [ 3154.170633] Key type .llcrypt registered [ 3154.288949] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3155.457031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3155.470486] alg: No test for adler32 (adler32-zlib) [ 3156.552721] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 3156.762390] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3158.399214] Key type lgssc registered [ 3159.741927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3167.923881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3167.959371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3169.408097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3174.859901] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3180.003576] LustreError: 110858:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3180.028975] LustreError: 110858:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3184.025145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3184.379665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3188.309106] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3194.708626] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3199.308287] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3205.411714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3205.868111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3212.043757] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3215.084600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3219.108967] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3224.511596] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3228.267588] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3235.812189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3235.814284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3235.835491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3235.872970] Lustre: Skipped 1 previous similar message [ 3238.473615] Lustre: server umount lustre-OST0000 complete [ 3246.049294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3246.054700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3246.067516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3246.086156] Lustre: Skipped 2 previous similar messages [ 3246.098434] Lustre: Skipped 1 previous similar message [ 3249.958447] Lustre: server umount lustre-MDT0000 complete [ 3251.168396] LustreError: 110855:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3251.180857] LustreError: 110855:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3252.881481] LustreError: 110839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853971 with bad export cookie 8113979911636875826 [ 3252.882879] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3252.898282] LustreError: 110839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3253.216431] Lustre: server umount lustre-MDT0001 complete [ 3263.678937] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3266.258916] Key type lgssc unregistered [ 3266.542768] LNet: 114498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3266.554645] LNetError: 114498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3266.580876] LNet: Removed LNI 192.168.203.116@tcp [ 3267.902474] Key type .llcrypt unregistered [ 3267.904737] Key type ._llcrypt unregistered [ 3290.792116] Key type ._llcrypt registered [ 3290.799758] Key type .llcrypt registered [ 3290.936629] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3292.153850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3292.167679] alg: No test for adler32 (adler32-zlib) [ 3293.314655] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 3293.857308] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3295.595411] Key type lgssc registered [ 3296.845596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3304.674765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3304.688714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3306.212979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3310.766522] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3316.720988] LustreError: 115716:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3316.748031] LustreError: 115716:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3319.330342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3319.882779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3324.917403] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3334.691285] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3342.558946] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3352.926675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3353.351607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3360.486256] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3369.787380] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3376.954880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3388.912196] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3388.919471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3388.956358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3390.432804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3390.444869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3390.472617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3392.486365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3392.500711] Lustre: Skipped 1 previous similar message [ 3393.941825] Lustre: server umount lustre-OST0000 complete [ 3401.708714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3401.724345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3401.747878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3404.370556] Lustre: server umount lustre-MDT0000 complete [ 3407.851806] LustreError: 116698:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3407.882784] LustreError: 116698:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3408.678882] LustreError: 117561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854127 with bad export cookie 12344446327299202121 [ 3408.682314] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3408.693491] LustreError: 117561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3409.268783] Lustre: server umount lustre-MDT0001 complete [ 3419.525342] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3421.875647] Key type lgssc unregistered [ 3422.180569] LNet: 119272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3422.187320] LNetError: 119272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3422.208175] LNet: Removed LNI 192.168.203.116@tcp [ 3422.734674] Key type .llcrypt unregistered [ 3422.737211] Key type ._llcrypt unregistered [ 3447.019307] Key type ._llcrypt registered [ 3447.023912] Key type .llcrypt registered [ 3447.319269] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3449.352551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3449.478071] alg: No test for adler32 (adler32-zlib) [ 3450.911704] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 3451.542345] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3453.417882] Key type lgssc registered [ 3455.368695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3465.044387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3465.055837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3466.383469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3470.356382] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3471.841705] LustreError: 120684:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3471.853492] LustreError: 120684:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3476.968948] LustreError: 120685:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3479.562371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3479.865803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3484.629547] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3492.929974] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3500.390100] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3509.913502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3510.369094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3511.449673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3517.291336] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3524.689892] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3531.683791] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3535.460437] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3535.471206] Lustre: Skipped 2 previous similar messages [ 3539.553390] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3539.608302] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3540.117885] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3540.121444] Lustre: Skipped 2 previous similar messages [ 3545.681402] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3545.685151] Lustre: Skipped 1 previous similar message [ 3550.840225] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.16@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3550.869057] Lustre: Skipped 2 previous similar messages [ 3553.575303] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.16@tcp 192.168.0.[2,10]@tcp [ 3563.647226] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.16/24@tcp in log lustre-MDT0000 [ 3563.658871] Lustre: Skipped 2 previous similar messages [ 3571.328485] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.16/24@tcp [ 3571.334830] Lustre: Skipped 1 previous similar message [ 3576.288430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3576.292164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3576.305413] Lustre: Skipped 1 previous similar message [ 3576.319219] Lustre: Skipped 1 previous similar message [ 3578.414368] Lustre: server umount lustre-OST0000 complete [ 3582.436103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3582.457981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3582.479097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3586.531137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3586.532641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3588.002935] Lustre: server umount lustre-MDT0000 complete [ 3591.435975] LustreError: 122529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854310 with bad export cookie 16572511553552358678 [ 3591.449356] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3591.459227] LustreError: 122529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3591.836075] Lustre: server umount lustre-MDT0001 complete [ 3603.043749] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3606.430919] Key type lgssc unregistered [ 3606.990840] LNet: 125678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3607.004476] LNetError: 125678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3607.038149] LNet: Removed LNI 192.168.203.116@tcp [ 3608.154937] Key type .llcrypt unregistered [ 3608.157063] Key type ._llcrypt unregistered [ 3639.972771] Key type ._llcrypt registered [ 3639.980489] Key type .llcrypt registered [ 3640.106546] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3641.828878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3641.900056] alg: No test for adler32 (adler32-zlib) [ 3642.882542] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 3643.201064] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3644.889159] Key type lgssc registered [ 3647.032747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3657.751664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3657.770000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3659.522704] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3659.542382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3659.552319] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.16/24@tcp [ 3665.139242] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3669.988192] LustreError: 127237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3670.001242] LustreError: 127237:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3674.158084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3674.557502] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3674.596174] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.16/24@tcp [ 3674.665178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3679.361662] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3686.615652] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3692.946343] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3702.005506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3702.777385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3704.893221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3713.870411] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3723.988874] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3734.510837] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3734.515413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3734.537420] Lustre: Skipped 1 previous similar message [ 3734.539075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3734.559017] Lustre: Skipped 1 previous similar message [ 3739.619982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3739.782948] Lustre: server umount lustre-OST0000 complete [ 3744.741321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3744.743192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3744.775657] Lustre: Skipped 1 previous similar message [ 3744.788456] Lustre: Skipped 2 previous similar messages [ 3749.860312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3749.864926] Lustre: Skipped 1 previous similar message [ 3750.529859] Lustre: server umount lustre-MDT0000 complete [ 3754.857189] LustreError: 129084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854473 with bad export cookie 12478777788704861460 [ 3754.860410] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3754.869215] LustreError: 129084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3755.338452] Lustre: server umount lustre-MDT0001 complete [ 3771.441854] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3775.181682] Key type lgssc unregistered [ 3775.675513] LNet: 130834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3775.681938] LNetError: 130834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3775.699779] LNet: Removed LNI 192.168.203.116@tcp [ 3776.834777] Key type .llcrypt unregistered [ 3776.837296] Key type ._llcrypt unregistered [ 3808.496363] Key type ._llcrypt registered [ 3808.501544] Key type .llcrypt registered [ 3808.797617] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 3828.814551] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3830.924231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3830.966106] alg: No test for adler32 (adler32-zlib) [ 3832.151468] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 3832.594915] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3834.399272] Key type lgssc registered [ 3836.068347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3872.107980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3872.143128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3873.411956] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3873.502318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3873.653212] Lustre: lustre-MDT0000: new disk, initializing [ 3873.739767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3873.776182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3878.430955] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3888.515225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3888.663340] Lustre: 134233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3888.765529] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3888.771987] Lustre: Skipped 1 previous similar message [ 3888.909536] Lustre: lustre-MDT0001: new disk, initializing [ 3889.034603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3889.117797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3889.134551] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3894.685511] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3906.478852] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3912.103487] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3918.911818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3919.178422] Lustre: lustre-OST0000: new disk, initializing [ 3919.183198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3919.192395] Lustre: 136206:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3919.271536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3924.901651] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3928.116525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3928.133274] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3928.279799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3934.714622] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3940.307695] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3940.525440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3946.220912] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3946.507693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3954.150084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3954.156733] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3954.173536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3954.197216] Lustre: Skipped 1 previous similar message [ 3959.278089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3959.289431] Lustre: Skipped 2 previous similar messages [ 3963.359169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3963.551688] Lustre: server umount lustre-OST0000 complete [ 3969.514943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3969.521541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3969.552978] Lustre: Skipped 1 previous similar message [ 3974.624757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3974.631838] Lustre: Skipped 2 previous similar messages [ 3974.810873] LustreError: 137746:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000a437ce2f) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3974.906296] Lustre: server umount lustre-MDT0000 complete [ 3978.490560] LustreError: 136205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854697 with bad export cookie 12846084150137292220 [ 3978.492127] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3978.499867] LustreError: 136205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3978.897672] Lustre: server umount lustre-MDT0001 complete [ 3987.121679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3987.922810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3992.870917] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3993.063110] LustreError: 138426:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3993.088905] LustreError: 138426:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3997.157474] LustreError: 138425:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4001.485445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4001.840424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4006.675153] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4014.749160] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4021.016711] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4028.894191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4029.313320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4034.517577] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4041.422436] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4046.197732] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4046.478771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4050.833936] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4051.055459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4060.650451] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4060.837968] Lustre: lustre-OST0001: new disk, initializing [ 4060.843671] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4060.848882] Lustre: 141798:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4060.943983] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4066.868683] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4068.963079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4068.977511] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4069.056186] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4075.818700] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4076.025230] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4081.809441] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4082.207794] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4090.722432] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95e6113b9800.ost_server_uuid 50 [ 4093.281695] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff95e6113b9800.ost_server_uuid in IDLE state after 0 sec [ 4105.193725] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4105.213270] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4105.219168] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4105.227288] Lustre: Skipped 1 previous similar message [ 4108.983873] Lustre: server umount lustre-OST0001 complete [ 4110.317986] LustreError: 140247:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4110.342126] LustreError: 140247:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4113.377376] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4113.387229] LustreError: Skipped 1 previous similar message [ 4113.393095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4113.406317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4113.411819] Lustre: Skipped 1 previous similar message [ 4115.425377] LustreError: 142263:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4115.450374] LustreError: 142263:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4118.793680] Lustre: server umount lustre-OST0000 complete [ 4124.645792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4124.663305] LustreError: Skipped 1 previous similar message [ 4124.665460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4124.691681] Lustre: Skipped 1 previous similar message [ 4128.555819] Lustre: server umount lustre-MDT0000 complete [ 4130.787486] LustreError: 140471:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4130.809932] LustreError: 140471:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4133.113813] LustreError: 138407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854851 with bad export cookie 12846084150137293102 [ 4133.128470] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4133.618875] Lustre: server umount lustre-MDT0001 complete [ 4159.599305] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 4162.449379] Key type lgssc unregistered [ 4162.813925] LNet: 144803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4162.820902] LNetError: 144803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4162.836752] LNet: Removed LNI 192.168.203.116@tcp [ 4163.825484] Key type .llcrypt unregistered [ 4163.831870] Key type ._llcrypt unregistered [ 4208.317204] Key type ._llcrypt registered [ 4208.320568] Key type .llcrypt registered [ 4208.419792] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 4222.161799] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4223.406098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4223.425898] alg: No test for adler32 (adler32-zlib) [ 4224.429294] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 4224.673604] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4226.343182] Key type lgssc registered [ 4227.265900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4261.250262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4261.318480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4262.569672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4262.624461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4262.699987] Lustre: lustre-MDT0000: new disk, initializing [ 4262.787266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4262.828208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4267.502515] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4279.027406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4279.141451] Lustre: 148956:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4279.202837] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4279.210877] Lustre: Skipped 1 previous similar message [ 4279.307184] Lustre: lustre-MDT0001: new disk, initializing [ 4279.400061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4279.460117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4279.481737] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4283.903674] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4294.199817] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4299.597536] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4306.547815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4306.785870] Lustre: lustre-OST0000: new disk, initializing [ 4306.790183] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4306.795201] Lustre: 150924:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4306.896889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4313.549770] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4314.206946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4314.229696] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4314.437725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4323.857565] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4328.944855] Lustre: Failing over lustre-OST0000 [ 4329.089552] Lustre: server umount lustre-OST0000 complete [ 4345.313643] Lustre: 147178:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788855048/real 1788855048] req@00000000f698ebae x1875750360745344/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788855064 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4345.388300] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4346.334980] Lustre: 147179:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788855048/real 1788855048] req@00000000873da693 x1875750360745472/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788855064 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4346.355105] Lustre: 147179:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4346.367758] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4348.641290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4349.078381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4349.098939] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4351.091166] Lustre: 147180:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788855053/real 1788855053] req@000000007efa9ecf x1875750360746624/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788855069 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4351.152174] Lustre: 147180:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4355.172551] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4357.234434] Lustre: 147178:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788855059/real 1788855059] req@00000000f0b98d20 x1875750360747776/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788855075 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4357.261920] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4358.538511] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4358.539122] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4361.312191] Lustre: 147177:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788855064/real 1788855064] req@00000000e2ef7118 x1875750360748928/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788855080 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4361.351807] Lustre: 147177:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4363.149950] Lustre: Failing over lustre-MDT0000 [ 4363.393870] Lustre: server umount lustre-MDT0000 complete [ 4365.286425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4365.299704] Lustre: Skipped 1 previous similar message [ 4368.758417] LustreError: 151209:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4368.811190] LustreError: 151209:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4370.401597] LustreError: 148964:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4370.417074] LustreError: 148964:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4373.833903] LustreError: 148963:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4378.947249] LustreError: 150517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4378.962470] LustreError: 150517:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4381.668951] Lustre: 147180:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788855084/real 1788855084] req@00000000d0f61573 x1875750360761984/t0(0) o400->MGC192.168.203.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788855100 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4381.735240] Lustre: 147180:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4381.756728] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4382.822891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4384.070724] LustreError: 148962:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4384.106253] LustreError: 148962:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4391.905094] LustreError: 147176:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000444667c7 x1875750360767488/t0(0) o250->MGC192.168.203.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4392.415571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4392.531353] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4395.335958] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4397.593133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4397.614542] Lustre: Skipped 1 previous similar message [ 4397.646280] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4397.745421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4398.169576] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4407.794417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4407.801345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4407.825547] Lustre: Skipped 2 previous similar messages [ 4412.596211] Lustre: server umount lustre-OST0000 complete [ 4418.016112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4418.020882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4418.047614] Lustre: Skipped 1 previous similar message [ 4418.060217] Lustre: Skipped 1 previous similar message [ 4423.140938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4423.152376] Lustre: Skipped 2 previous similar messages [ 4425.516179] Lustre: server umount lustre-MDT0000 complete [ 4427.237491] LustreError: 150517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4427.269571] LustreError: 150517:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 4428.996537] LustreError: 148949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855147 with bad export cookie 17769388623981219686 [ 4429.004486] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4429.029802] LustreError: 148949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4429.444493] Lustre: server umount lustre-MDT0001 complete [ 4439.463115] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 4442.454885] Key type lgssc unregistered [ 4442.770764] LNet: 154581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4442.783405] LNetError: 154581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4442.807505] LNet: Removed LNI 192.168.203.116@tcp [ 4443.906961] Key type .llcrypt unregistered [ 4443.908635] Key type ._llcrypt unregistered [ 4478.879653] Key type ._llcrypt registered [ 4478.881690] Key type .llcrypt registered [ 4478.991737] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4479.962538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4479.981120] alg: No test for adler32 (adler32-zlib) [ 4480.972438] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 4481.151661] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4482.799203] Key type lgssc registered [ 4483.875465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4490.818714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4490.848532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4492.310621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4496.454673] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4502.508434] LustreError: 156098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4502.521021] LustreError: 156098:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4505.600760] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4507.618830] LustreError: 156097:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4509.336349] LustreError: 156126:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4509.345874] LustreError: 156126:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4509.354884] LustreError: 156126:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4509.631652] Lustre: server umount lustre-MDT0000 complete [ 4526.821962] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 4540.756435] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4583.767281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4584.109204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4584.137681] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4584.206978] Lustre: lustre-MDT0000: new disk, initializing [ 4584.283832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4584.299801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4588.796168] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4599.172462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4599.296240] Lustre: 159431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4599.344579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4599.350533] Lustre: Skipped 1 previous similar message [ 4599.422787] Lustre: lustre-MDT0001: new disk, initializing [ 4599.575794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4599.647425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4599.655085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4604.724736] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4613.852736] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4619.056640] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4625.693184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4625.945800] Lustre: lustre-OST0000: new disk, initializing [ 4625.957555] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4625.972517] Lustre: 161398:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4626.072468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4632.193399] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4632.641646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4632.657664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4632.794254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4642.199748] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4646.680236] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4648.369154] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 04:16:05 (1788855365) [ 4881.375901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4881.391606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4881.417164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4883.940099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4883.941590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4883.985195] Lustre: Skipped 2 previous similar messages [ 4887.498635] Lustre: server umount lustre-MDT0000 complete [ 4889.055680] LustreError: 159437:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4889.082477] LustreError: 159437:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4890.515632] LustreError: 160233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855609 with bad export cookie 8791978083269119025 [ 4890.517251] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4890.534327] LustreError: 160233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4894.182195] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4894.185246] LustreError: 163166:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4894.199857] Lustre: Skipped 1 previous similar message [ 4894.226226] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4896.896651] Lustre: server umount lustre-MDT0001 complete [ 4906.866157] Lustre: server umount lustre-OST0000 complete [ 4916.815788] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 4928.957458] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4970.261722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4970.836262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4970.923908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4971.145822] Lustre: lustre-MDT0000: new disk, initializing [ 4971.322592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4971.374918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4979.000479] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4992.274117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4992.449391] Lustre: 166054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4992.522258] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4992.527469] Lustre: Skipped 1 previous similar message [ 4992.650250] Lustre: lustre-MDT0001: new disk, initializing [ 4992.766206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4992.851868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4992.870139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4998.024858] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5007.736194] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5012.713301] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5019.756696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5020.132340] Lustre: lustre-OST0000: new disk, initializing [ 5020.141822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5020.151986] Lustre: 168025:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5020.251108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5026.564655] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5028.425155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5028.437471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5028.546448] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5036.784701] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5041.126092] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5041.347993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5046.200627] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5046.582521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5049.324109] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5049.339659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5049.343791] LustreError: Skipped 1 previous similar message [ 5049.344769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5049.344773] Lustre: Skipped 1 previous similar message [ 5049.401202] Lustre: Skipped 1 previous similar message [ 5054.445590] LustreError: 168009:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5054.475530] LustreError: 168009:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5054.592225] Lustre: server umount lustre-OST0000 complete [ 5059.552202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5059.556728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5059.588586] Lustre: Skipped 1 previous similar message [ 5059.614107] Lustre: Skipped 2 previous similar messages [ 5065.739305] Lustre: server umount lustre-MDT0000 complete [ 5069.794986] LustreError: 167565:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5069.827843] LustreError: 167565:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5069.856464] LustreError: 166047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855788 with bad export cookie 8791978083269277855 [ 5069.863569] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5069.871888] LustreError: 166047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5070.561529] Lustre: server umount lustre-MDT0001 complete [ 5080.430507] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:23:17 (1788855797) [ 5099.119979] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 5109.772746] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5143.558515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5143.817942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5143.857394] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5143.950157] Lustre: lustre-MDT0000: new disk, initializing [ 5144.068732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5144.083204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5150.072174] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5163.032043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5163.183654] Lustre: 172781:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5163.590637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5168.669515] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5178.763059] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5183.941418] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5190.099331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5190.405397] Lustre: lustre-OST0000: new disk, initializing [ 5190.408711] Lustre: Skipped 1 previous similar message [ 5190.413989] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5190.421088] Lustre: Skipped 2 previous similar messages [ 5190.427112] Lustre: 174757:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5196.965786] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5199.933706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5199.953946] Lustre: Skipped 1 previous similar message [ 5199.956132] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5200.028463] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5206.233725] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5215.318478] Lustre: server umount lustre-OST0000 complete [ 5219.808357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5219.823683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5219.841050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5219.848945] Lustre: Skipped 2 previous similar messages [ 5229.211641] Lustre: server umount lustre-MDT0000 complete [ 5231.072903] LustreError: 172787:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5231.099944] LustreError: 172787:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5232.095882] Lustre: 155408:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788855934/real 1788855934] req@00000000a719cbd1 x1875750630957952/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788855950 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5233.720556] LustreError: 172772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855952 with bad export cookie 8791978083269278737 [ 5233.722773] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5233.740494] LustreError: 172772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5234.020479] Lustre: server umount lustre-MDT0001 complete [ 5243.505135] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:26:00 (1788855960) [ 5260.132808] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 5270.015551] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5305.700538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5306.036193] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5306.091548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5306.233629] Lustre: lustre-MDT0000: new disk, initializing [ 5306.350624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5306.360890] Lustre: Skipped 2 previous similar messages [ 5306.382024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5311.175817] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5322.806807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5322.984405] Lustre: 179080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5323.536301] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5330.838368] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5348.177055] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5355.300757] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5363.246411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5363.496415] Lustre: 181058:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5370.559798] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5372.526636] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5372.625702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5380.631744] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5388.258514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5388.274418] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5388.276491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5388.276502] Lustre: Skipped 4 previous similar messages [ 5388.291504] Lustre: Skipped 3 previous similar messages [ 5393.835315] Lustre: server umount lustre-OST0000 complete [ 5412.319148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5412.572358] Lustre: server umount lustre-MDT0000 complete [ 5413.857693] LustreError: 180455:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5413.885953] LustreError: 180455:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5416.167506] LustreError: 181069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856135 with bad export cookie 8791978083269279822 [ 5416.177638] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5416.191237] LustreError: 181069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5416.600856] Lustre: server umount lustre-MDT0001 complete [ 5427.616233] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5430.878951] Key type lgssc unregistered [ 5431.320326] LNet: 182913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5431.335383] LNetError: 182913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5431.363284] LNet: Removed LNI 192.168.203.116@tcp [ 5432.279913] Key type .llcrypt unregistered [ 5432.281930] Key type ._llcrypt unregistered [ 5457.252954] Key type ._llcrypt registered [ 5457.256814] Key type .llcrypt registered [ 5457.358603] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5458.376401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5458.609969] alg: No test for adler32 (adler32-zlib) [ 5459.731578] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 5459.951416] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5461.575210] Key type lgssc registered [ 5462.955472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5471.734372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5471.779879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5473.350084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5479.126859] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5483.497547] LustreError: 184327:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5483.532858] LustreError: 184327:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5488.612162] LustreError: 184326:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5489.528037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5489.958724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5494.287338] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5502.389618] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5508.541029] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5518.856162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5519.420632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5527.146952] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5535.238432] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5543.992397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5564.390286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5564.395070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5564.401463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5564.410382] Lustre: Skipped 1 previous similar message [ 5564.435840] Lustre: Skipped 1 previous similar message [ 5569.154063] Lustre: server umount lustre-OST0000 complete [ 5574.626689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5574.637858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5574.646862] Lustre: Skipped 1 previous similar message [ 5579.249946] Lustre: server umount lustre-MDT0000 complete [ 5579.750016] LustreError: 184326:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5579.775042] LustreError: 184326:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5582.626358] LustreError: 184308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856301 with bad export cookie 11466107173386460466 [ 5582.632950] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5582.647239] LustreError: 184308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5583.075538] Lustre: server umount lustre-MDT0001 complete [ 5593.538918] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 5596.448937] Key type lgssc unregistered [ 5596.773097] LNet: 187884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5596.780887] LNetError: 187884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5596.793432] LNet: Removed LNI 192.168.203.116@tcp [ 5597.580824] Key type .llcrypt unregistered [ 5597.583352] Key type ._llcrypt unregistered [ 5619.677848] Key type ._llcrypt registered [ 5619.680401] Key type .llcrypt registered [ 5619.781724] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 5620.962499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5621.006098] alg: No test for adler32 (adler32-zlib) [ 5622.100789] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 5622.365857] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 5624.079241] Key type lgssc registered [ 5625.128541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5632.395919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5632.406751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5633.919918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5638.593079] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5643.246964] LustreError: 189297:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5643.268342] LustreError: 189297:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5647.733551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5648.291699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5652.093583] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5660.211734] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5666.410362] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5673.099127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5673.527390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5678.648886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5680.817879] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5689.561278] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5699.041420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5699.067652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5699.075954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5699.116123] Lustre: Skipped 1 previous similar message [ 5703.073555] Lustre: lustre-OST0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 5703.092879] Lustre: Skipped 1 previous similar message [ 5704.178613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5705.131598] Lustre: server umount lustre-OST0000 complete [ 5717.245639] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff95e620580000.ost_server_uuid 50 [ 5720.079348] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5735.254078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5735.288613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5735.288631] Lustre: Skipped 1 previous similar message [ 5735.310715] Lustre: Skipped 1 previous similar message [ 5740.516069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5740.524444] Lustre: Skipped 2 previous similar messages [ 5741.679505] Lustre: server umount lustre-MDT0000 complete [ 5745.633877] LustreError: 189297:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5745.651098] LustreError: 189297:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5746.744978] LustreError: 189278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856465 with bad export cookie 4796375726183915308 [ 5746.753549] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5746.761354] LustreError: 189278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5747.591354] Lustre: server umount lustre-MDT0001 complete [ 5759.966326] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 04:34:36 (1788856476) [ 5768.559923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5769.572139] LustreError: 193213:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5769.724202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5774.806513] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5774.817838] LustreError: 193214:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5779.937254] LustreError: 193213:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5783.227573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5783.545096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5788.703856] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5795.864449] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5801.755792] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5808.875233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5809.076933] LustreError: 195034:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5809.233316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5815.297059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5815.865362] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5824.393714] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5832.838071] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5833.074711] Lustre: lustre-OST0001: new disk, initializing [ 5833.079759] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5833.086491] Lustre: 196055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5833.210300] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5840.345737] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5841.523933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5841.537063] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5841.725053] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5851.425410] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5855.203956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5855.218184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5855.231031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5856.224306] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5860.464310] Lustre: server umount lustre-OST0000 complete [ 5862.218590] LustreError: 195035:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5866.513621] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5866.702664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5871.796611] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5872.047435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5887.970036] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5887.988744] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5887.995047] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5888.009351] Lustre: Skipped 2 previous similar messages [ 5888.020925] Lustre: Skipped 4 previous similar messages [ 5892.136074] Lustre: server umount lustre-OST0001 complete [ 5898.211310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5902.754324] Lustre: server umount lustre-MDT0000 complete [ 5903.331317] LustreError: 193214:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5903.358631] LustreError: 193214:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 5906.729479] LustreError: 193195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856625 with bad export cookie 4796375726183916379 [ 5906.733228] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5906.740164] LustreError: 193195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5907.177670] Lustre: server umount lustre-MDT0001 complete [ 5931.043471] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 04:37:28 (1788856648) [ 5937.987671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5938.160785] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5938.215258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5938.357419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5942.960768] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5950.567155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5950.662450] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5950.734916] Lustre: 199498:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5955.483341] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5962.996902] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5968.530675] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5976.413175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5976.679162] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5983.349922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5983.747829] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 5994.969797] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6004.104075] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6004.254873] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6004.590399] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6004.594560] Lustre: Skipped 2 previous similar messages [ 6012.525879] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6013.636075] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6019.582690] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6024.164923] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6024.180181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6024.181777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6024.181784] Lustre: Skipped 3 previous similar messages [ 6024.187643] LustreError: Skipped 1 previous similar message [ 6024.212187] Lustre: Skipped 2 previous similar messages [ 6028.440232] Lustre: server umount lustre-OST0000 complete [ 6029.296036] LustreError: 201346:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6029.315752] LustreError: 201346:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6044.640977] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6044.641110] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6044.658723] LustreError: Skipped 1 previous similar message [ 6044.686256] Lustre: Skipped 1 previous similar message [ 6050.235553] Lustre: server umount lustre-OST0001 complete [ 6068.191415] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6068.468763] Lustre: server umount lustre-MDT0000 complete [ 6072.268079] LustreError: 201368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856791 with bad export cookie 4796375726183917786 [ 6072.272329] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6072.296757] LustreError: 201368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6072.866135] Lustre: server umount lustre-MDT0001 complete [ 6098.422610] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 04:40:15 (1788856815) [ 6115.557367] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 6129.483649] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6171.017585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6171.318342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6171.372724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6171.510273] Lustre: lustre-MDT0000: new disk, initializing [ 6171.703643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6171.767935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6176.771543] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6188.831374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6189.099557] Lustre: 207594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6189.192821] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6189.198704] Lustre: Skipped 1 previous similar message [ 6189.335667] Lustre: lustre-MDT0001: new disk, initializing [ 6189.521402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6189.537380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6195.746848] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6206.823603] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6213.827377] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6221.113250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6221.451416] Lustre: lustre-OST0000: new disk, initializing [ 6221.454811] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6221.461601] Lustre: 209566:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6228.067184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6228.087043] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6228.235298] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6228.517818] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6238.062375] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6242.237872] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6242.455818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6248.347592] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6248.682793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6254.048150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6254.063073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6254.064163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6254.064168] Lustre: Skipped 12 previous similar messages [ 6254.104119] Lustre: Skipped 3 previous similar messages [ 6257.518107] Lustre: server umount lustre-OST0000 complete [ 6267.862402] Lustre: server umount lustre-MDT0000 complete [ 6269.410571] LustreError: 207604:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6269.435644] LustreError: 207604:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [ 6271.910296] LustreError: 207587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856990 with bad export cookie 4796375726183919571 [ 6271.910976] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6271.931681] LustreError: 207587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6272.322448] Lustre: server umount lustre-MDT0001 complete [ 6280.863323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6286.664902] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6295.188342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6300.050031] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6309.769054] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6316.267111] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6323.828898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6331.099349] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6339.141936] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6344.108791] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6344.336953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6349.052655] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6349.304783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6351.329052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6351.332767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6351.349822] Lustre: Skipped 2 previous similar messages [ 6357.577422] Lustre: server umount lustre-OST0000 complete [ 6364.546743] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6369.280044] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6398.866821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6406.484412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.506251] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.516244] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6406.520134] Lustre: Skipped 1 previous similar message [ 6406.931309] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6418.105597] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6420.319190] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6427.115275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6427.117580] Lustre: Skipped 7 previous similar messages [ 6432.080413] Lustre: server umount lustre-OST0000 complete [ 6442.753687] Lustre: server umount lustre-MDT0000 complete [ 6448.245963] LustreError: 211764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857167 with bad export cookie 4796375726183920453 [ 6448.257048] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6448.280474] LustreError: 211764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6449.077687] Lustre: server umount lustre-MDT0001 complete [ 6459.059885] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 04:46:16 (1788857176) [ 6466.931854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6467.664802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6467.674781] Lustre: Skipped 6 previous similar messages [ 6471.539945] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6484.355637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6491.192937] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6502.133552] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6510.214640] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6517.350778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6523.643803] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6531.376285] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6535.918072] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6539.361413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6543.659895] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6543.886854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6549.899167] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6550.140812] Lustre: lustre-OST0001: new disk, initializing [ 6550.157131] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6550.163691] Lustre: 221151:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6556.669536] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6559.908579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6559.917710] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6560.056592] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6566.587311] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6571.442936] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6571.661989] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6576.820977] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6577.114916] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6580.712426] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6580.718576] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6580.726499] LustreError: Skipped 1 previous similar message [ 6580.745743] Lustre: Skipped 6 previous similar messages [ 6585.601940] Lustre: server umount lustre-OST0001 complete [ 6585.825547] LustreError: 219579:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6585.845210] LustreError: 219579:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 6592.377663] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6592.743952] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6598.419777] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6598.725996] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6628.485933] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6633.979257] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6633.989037] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6633.990364] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6634.015322] Lustre: Skipped 1 previous similar message [ 6636.245982] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6643.889237] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6645.219804] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6653.770800] Lustre: server umount lustre-OST0001 complete [ 6685.152648] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6685.159526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6685.199418] Lustre: Skipped 15 previous similar messages [ 6685.599698] Lustre: server umount lustre-OST0000 complete [ 6697.133198] Lustre: server umount lustre-MDT0000 complete [ 6701.059944] LustreError: 219603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857419 with bad export cookie 4796375726183921419 [ 6701.062017] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6701.078409] LustreError: 219603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6707.618459] Lustre: server umount lustre-MDT0001 complete [ 6734.443546] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 04:50:51 (1788857451) [ 6741.461229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6741.728425] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6741.755976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6746.907487] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6755.041543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6755.162536] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6755.187700] Lustre: 226397:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6759.785420] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6766.769136] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6772.165617] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6779.802961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6780.003276] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6787.164434] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6796.556937] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6806.574159] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6806.766959] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6814.021584] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6820.972711] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6822.529855] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6826.525580] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6826.736892] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6830.420415] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6830.632785] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6835.432805] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95e604ad2800.ost_server_uuid 50 [ 6837.154587] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff95e604ad2800.ost_server_uuid in IDLE state after 0 sec [ 6838.285352] Lustre: Permanently deactivating lustre-OST0001 [ 6838.294899] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6843.961637] Lustre: Permanently reactivating lustre-OST0001 [ 6843.997115] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6844.006254] Lustre: Skipped 2 previous similar messages [ 6850.032540] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6850.050483] LustreError: Skipped 5 previous similar messages [ 6850.059859] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6850.085228] Lustre: Skipped 6 previous similar messages [ 6855.372430] Lustre: server umount lustre-OST0001 complete [ 6864.962915] Lustre: server umount lustre-OST0000 complete [ 6883.295132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6883.551486] Lustre: server umount lustre-MDT0000 complete [ 6886.994372] LustreError: 226391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857605 with bad export cookie 4796375726183922707 [ 6886.997354] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6887.007376] LustreError: 226391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6887.332375] Lustre: server umount lustre-MDT0001 complete [ 6909.466825] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 04:53:46 (1788857626) [ 6918.522431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6918.955687] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6918.990627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6918.997474] Lustre: Skipped 2 previous similar messages [ 6923.644076] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6931.990710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6932.083737] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6932.099241] Lustre: 233099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6937.958572] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6946.490356] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6952.204560] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6959.111468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6959.484491] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6959.523682] Lustre: Permanently deactivating lustre-OST0000 [ 6959.541234] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6966.550599] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6969.360103] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6974.064639] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6980.911758] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6981.083731] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6981.299668] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6981.313890] Lustre: Skipped 11 previous similar messages [ 6988.192863] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6989.960916] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6996.758300] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6998.479608] Lustre: Permanently reactivating lustre-OST0000 [ 6998.506925] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6998.514485] Lustre: Skipped 2 previous similar messages [ 6999.973881] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6999.985734] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7000.001176] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7000.055219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7000.074757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7029.215827] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7029.370435] Lustre: server umount lustre-OST0001 complete [ 7039.680880] Lustre: server umount lustre-OST0000 complete [ 7049.829753] Lustre: server umount lustre-MDT0000 complete [ 7053.021732] LustreError: 234968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857771 with bad export cookie 4796375726183924373 [ 7053.030859] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7053.042372] LustreError: 234968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7053.506310] Lustre: server umount lustre-MDT0001 complete [ 7063.798349] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7066.548611] Key type lgssc unregistered [ 7066.884805] LNet: 237978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7066.901229] LNetError: 237978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7067.940684] LNet: Removed LNI 192.168.203.116@tcp [ 7068.665647] Key type .llcrypt unregistered [ 7068.669983] Key type ._llcrypt unregistered [ 7091.718722] Key type ._llcrypt registered [ 7091.721552] Key type .llcrypt registered [ 7091.857802] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7092.937120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7093.108104] alg: No test for adler32 (adler32-zlib) [ 7094.255764] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 7094.491083] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7096.167218] Key type lgssc registered [ 7097.487415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7108.088637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7108.104374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7109.559221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7113.519754] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7119.843819] LustreError: 239446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7119.869109] LustreError: 239446:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7121.678921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7121.884286] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7121.907847] Lustre: 239434:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7121.916614] Lustre: Permanently deactivating lustre-MDT0001 [ 7121.926830] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7122.270151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7126.216474] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7126.222351] LustreError: 239476:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7126.241245] LustreError: 239476:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7126.260170] LustreError: 239476:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 7126.672763] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7134.885305] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7140.113652] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7146.155097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7146.543267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7147.567716] LustreError: 241285:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7147.587354] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7152.410660] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7152.612333] LustreError: 241285:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7154.664907] LustreError: 241285:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7154.692514] LustreError: 241285:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7154.695907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7159.790495] LustreError: 241374:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7159.818281] LustreError: 241374:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7160.147332] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7168.039355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7168.234712] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7173.629537] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7173.636208] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7174.839481] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7181.533118] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7185.276384] Lustre: Permanently reactivating lustre-MDT0001 [ 7185.295317] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7185.300891] Lustre: Skipped 2 previous similar messages [ 7191.405279] Lustre: lustre-MDT0001: Client 97fb8e72-f9cb-4e7c-bf64-6e5bae3f4751 (at 192.168.203.16@tcp) reconnecting [ 7197.336661] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7198.757960] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7203.383318] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7203.567856] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7207.743727] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7207.953181] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7210.309329] Lustre: Permanently deactivating lustre-MDT0001 [ 7210.323159] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7210.328654] Lustre: Skipped 2 previous similar messages [ 7219.276257] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7229.928871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7229.944350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7229.962179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7231.977028] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7231.980119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7231.995819] Lustre: Skipped 2 previous similar messages [ 7232.008473] Lustre: Skipped 3 previous similar messages [ 7235.202215] Lustre: server umount lustre-MDT0000 complete [ 7237.088544] LustreError: 239446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7237.111767] LustreError: 239446:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 7238.754434] LustreError: 242992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857957 with bad export cookie 8310518002720751847 [ 7238.771480] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7239.191302] Lustre: server umount lustre-MDT0001 complete [ 7245.280467] Lustre: server umount lustre-OST0000 complete [ 7256.500976] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 04:59:33 (1788857973) [ 7265.151650] Lustre: server umount lustre-OST0001 complete [ 7271.434371] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 7280.002533] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7311.066547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7311.283443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7311.293586] Lustre: Skipped 2 previous similar messages [ 7311.322450] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7311.404443] Lustre: lustre-MDT0000: new disk, initializing [ 7311.496601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7311.510944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7315.054600] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7323.652406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7323.766348] Lustre: 247749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7323.842436] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7323.846724] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7323.848263] Lustre: Skipped 1 previous similar message [ 7323.925601] Lustre: lustre-MDT0001: new disk, initializing [ 7324.041637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7324.098134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7324.112242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7327.929444] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7337.092954] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7342.849847] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7349.655651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7349.939880] Lustre: lustre-OST0000: new disk, initializing [ 7349.944612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7349.951970] Lustre: 249727:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7350.047467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7357.776926] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7359.028581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7359.039970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7359.162829] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7367.036738] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7368.860490] LustreError: 250447:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7376.639661] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7376.785955] Lustre: lustre-OST0001: new disk, initializing [ 7376.791855] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7376.798859] Lustre: 250842:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7378.935076] LustreError: 250447:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7378.977196] LustreError: 247754:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7383.118097] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7384.692490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7384.703791] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7384.831763] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7389.027160] LustreError: 247754:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7391.668874] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7393.214822] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7395.304652] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7395.307018] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7395.348741] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7395.355880] Lustre: Skipped 1 previous similar message [ 7400.433882] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7400.441593] Lustre: Skipped 2 previous similar messages [ 7400.832436] Lustre: server umount lustre-OST0001 complete [ 7402.826699] LustreError: 249710:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7420.906585] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7420.915576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7420.917892] Lustre: Skipped 1 previous similar message [ 7420.922271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7420.928492] LustreError: Skipped 1 previous similar message [ 7420.950248] Lustre: Skipped 1 previous similar message [ 7423.189694] Lustre: server umount lustre-OST0000 complete [ 7426.530594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7426.541787] LustreError: Skipped 1 previous similar message [ 7426.549105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7426.565684] Lustre: Skipped 1 previous similar message [ 7432.693218] Lustre: server umount lustre-MDT0000 complete [ 7435.868390] LustreError: 248556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858154 with bad export cookie 8310518002720754661 [ 7435.874442] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7435.880665] LustreError: 248556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7442.415902] Lustre: server umount lustre-MDT0001 complete [ 7451.964279] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7454.457558] Key type lgssc unregistered [ 7454.739148] LNet: 252873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7454.751300] LNetError: 252873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7454.781657] LNet: Removed LNI 192.168.203.116@tcp [ 7455.507338] Key type .llcrypt unregistered [ 7455.509598] Key type ._llcrypt unregistered [ 7490.766554] Key type ._llcrypt registered [ 7490.768685] Key type .llcrypt registered [ 7490.913749] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7492.258335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7492.300163] alg: No test for adler32 (adler32-zlib) [ 7493.298094] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 7493.574389] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7495.375863] Key type lgssc registered [ 7497.051123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7505.802856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7505.831921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7507.202615] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7507.216709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7507.322189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7511.926709] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7520.574710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7520.759316] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7520.793921] Lustre: 255078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7521.280867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7526.067780] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7532.694767] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7537.574451] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7544.010415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7544.184738] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7544.362694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7550.211478] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7554.183095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7557.388354] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7605.734520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7605.742626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7605.762476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7610.359899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7610.371095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7610.387045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7611.782499] Lustre: server umount lustre-OST0000 complete [ 7616.620174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7629.512323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7629.838558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7630.889856] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7630.915391] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7636.019090] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7636.074762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7636.098239] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7642.839348] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7682.016867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7682.026114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7682.036161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7682.048692] Lustre: Skipped 1 previous similar message [ 7687.362693] Lustre: server umount lustre-OST0000 complete [ 7692.769188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7692.771734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7692.791505] Lustre: Skipped 2 previous similar messages [ 7692.813753] Lustre: Skipped 2 previous similar messages [ 7696.712073] Lustre: server umount lustre-MDT0000 complete [ 7700.267992] LustreError: 255821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858419 with bad export cookie 16059100471019180178 [ 7700.269944] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7700.281824] LustreError: 255821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7700.611518] Lustre: server umount lustre-MDT0001 complete [ 7710.421555] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7712.994678] Key type lgssc unregistered [ 7713.345850] LNet: 260502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7713.361112] LNetError: 260502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7714.406925] LNet: Removed LNI 192.168.203.116@tcp [ 7715.196408] Key type .llcrypt unregistered [ 7715.201347] Key type ._llcrypt unregistered [ 7737.438478] Key type ._llcrypt registered [ 7737.440866] Key type .llcrypt registered [ 7737.642245] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7739.268077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7739.322707] alg: No test for adler32 (adler32-zlib) [ 7740.496227] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 7741.064939] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7742.824680] Key type lgssc registered [ 7744.120273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7754.100293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7754.139352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7755.693357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7762.082772] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7765.997369] LustreError: 261916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7766.010218] LustreError: 261916:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7771.120653] LustreError: 261917:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7773.363736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7773.674367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7778.435743] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7786.491870] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7792.161649] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7798.745540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7799.173322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7800.239823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7805.006829] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7811.613436] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7844.322301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7844.325973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7844.334299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7844.355935] Lustre: Skipped 1 previous similar message [ 7848.706453] Lustre: server umount lustre-OST0000 complete [ 7854.562466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7854.579319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7854.581895] Lustre: Skipped 1 previous similar message [ 7854.590971] Lustre: Skipped 1 previous similar message [ 7858.959953] Lustre: server umount lustre-MDT0000 complete [ 7859.693576] LustreError: 261916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7859.711236] LustreError: 261916:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7862.552434] LustreError: 261898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858581 with bad export cookie 16268281631411589168 [ 7862.561494] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7862.562903] LustreError: 261898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7862.963106] Lustre: server umount lustre-MDT0001 complete [ 7873.911814] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7877.193281] Key type lgssc unregistered [ 7877.485203] LNet: 266242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7877.499326] LNetError: 266242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7877.515081] LNet: Removed LNI 192.168.203.116@tcp [ 7878.336966] Key type .llcrypt unregistered [ 7878.342880] Key type ._llcrypt unregistered [ 7896.293855] Key type ._llcrypt registered [ 7896.297552] Key type .llcrypt registered [ 7896.397758] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7897.487064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7897.609626] alg: No test for adler32 (adler32-zlib) [ 7898.686229] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 7898.961836] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7900.735201] Key type lgssc registered [ 7901.864473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7908.666120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7908.679900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7910.224747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7914.918663] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7920.614665] LustreError: 267460:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7920.633618] LustreError: 267460:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7923.408890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7923.773842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7928.903800] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7935.823354] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7940.674326] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7946.940531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7947.345381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7953.061427] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7955.578619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7960.564541] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7974.377658] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7974.383890] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7974.406242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7976.422475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7976.424796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7976.448853] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7977.780670] Lustre: server umount lustre-OST0000 complete [ 7981.537088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7981.545164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7981.566931] Lustre: Skipped 1 previous similar message [ 7981.574961] Lustre: Skipped 1 previous similar message [ 7986.657133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7986.665113] Lustre: Skipped 2 previous similar messages [ 7991.783833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7991.797733] Lustre: Skipped 1 previous similar message [ 7995.359148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7995.556383] Lustre: server umount lustre-MDT0000 complete [ 7996.897976] LustreError: 267460:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7996.910731] LustreError: 267460:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7999.516503] LustreError: 267440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858718 with bad export cookie 13149888127110909406 [ 7999.518914] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7999.531799] LustreError: 267440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7999.790147] Lustre: server umount lustre-MDT0001 complete [ 8010.328213] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 8013.042539] Key type lgssc unregistered [ 8013.379028] LNet: 271139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8013.404046] LNetError: 271139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8013.423562] LNet: Removed LNI 192.168.203.116@tcp [ 8014.464443] Key type .llcrypt unregistered [ 8014.467092] Key type ._llcrypt unregistered [ 8041.465815] Key type ._llcrypt registered [ 8041.470415] Key type .llcrypt registered [ 8041.665264] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8042.893245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8042.943131] alg: No test for adler32 (adler32-zlib) [ 8044.129705] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 8044.482587] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 8046.183188] Key type lgssc registered [ 8047.270374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8054.552684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8054.577303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8056.120387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8060.907451] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8065.506372] LustreError: 272563:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8065.528818] LustreError: 272563:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8070.568907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8070.626320] LustreError: 272562:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8070.956375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8075.699639] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8083.612170] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8089.517980] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8096.957834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8097.391466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8100.478610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8104.726956] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8112.853515] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8146.913565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8146.915630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8146.924501] LustreError: Skipped 1 previous similar message [ 8146.927562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8146.945883] Lustre: Skipped 1 previous similar message [ 8146.964599] Lustre: Skipped 1 previous similar message [ 8148.407888] Lustre: server umount lustre-OST0000 complete [ 8153.058180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8153.065212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8153.085307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8157.159655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8157.161103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8158.435110] Lustre: server umount lustre-MDT0000 complete [ 8161.684613] LustreError: 272541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858880 with bad export cookie 3272742419698064092 [ 8161.693519] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8161.704313] LustreError: 272541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8161.989683] Lustre: server umount lustre-MDT0001 complete [ 8173.335192] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 8176.311183] Key type lgssc unregistered [ 8176.555510] LNet: 276868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8176.567337] LNetError: 276868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8176.581649] LNet: Removed LNI 192.168.203.116@tcp [ 8177.422888] Key type .llcrypt unregistered [ 8177.432069] Key type ._llcrypt unregistered [ 8196.967545] Key type ._llcrypt registered [ 8196.969103] Key type .llcrypt registered [ 8197.097225] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8198.042620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8198.182230] alg: No test for adler32 (adler32-zlib) [ 8199.325940] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 8199.554598] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 8201.207279] Key type lgssc registered [ 8202.251720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8210.183747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8210.198728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8211.695219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8216.038930] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8221.161755] LustreError: 278102:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8221.191477] LustreError: 278102:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8224.821831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8225.290511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8229.753873] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8238.026684] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8244.404622] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8252.480841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8253.223452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8256.313678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8260.891460] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8268.555400] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8286.184906] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8286.194564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8286.200470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8286.220210] Lustre: Skipped 1 previous similar message [ 8288.918126] Lustre: server umount lustre-OST0000 complete [ 8296.418334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8296.450674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8296.454831] Lustre: Skipped 2 previous similar messages [ 8299.632076] Lustre: server umount lustre-MDT0000 complete [ 8301.537864] LustreError: 278103:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8301.558594] LustreError: 278103:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8303.400142] LustreError: 278069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859022 with bad export cookie 16229442897557961527 [ 8303.404980] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8303.410933] LustreError: 278069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8304.071191] Lustre: server umount lustre-MDT0001 complete [ 8315.338301] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 8317.943678] Key type lgssc unregistered [ 8318.252595] LNet: 281801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8318.265766] LNetError: 281801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8318.279267] LNet: Removed LNI 192.168.203.116@tcp [ 8318.989683] Key type .llcrypt unregistered [ 8318.994713] Key type ._llcrypt unregistered [ 8330.266330] Key type ._llcrypt registered [ 8330.268370] Key type .llcrypt registered [ 8330.427497] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8356.824836] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 8375.618233] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8377.410910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8377.548926] alg: No test for adler32 (adler32-zlib) [ 8378.638950] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 8378.816872] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 8380.519973] Key type lgssc registered [ 8381.607699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8417.880635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8417.927259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8419.150803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8419.195486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8419.314440] Lustre: lustre-MDT0000: new disk, initializing [ 8419.412242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8419.439510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8426.918429] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8440.097910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8440.331935] Lustre: 285352:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8440.497417] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8440.507784] Lustre: Skipped 1 previous similar message [ 8440.665480] Lustre: lustre-MDT0001: new disk, initializing [ 8440.846057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8440.942073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8440.965336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8446.164836] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8456.902884] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8464.388276] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8471.064967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8471.322438] Lustre: lustre-OST0000: new disk, initializing [ 8471.326511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8471.333584] Lustre: 287323:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8471.443472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8477.978323] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8478.806293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8478.818758] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8478.931951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8487.402449] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8492.222418] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8492.391893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8496.999266] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8497.270970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8499.687077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8499.689675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8499.691747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8499.777775] Lustre: Skipped 1 previous similar message [ 8504.812688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8504.828146] Lustre: Skipped 2 previous similar messages [ 8505.584134] Lustre: server umount lustre-OST0000 complete [ 8509.921183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8509.926664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8509.929567] Lustre: Skipped 1 previous similar message [ 8509.950169] Lustre: Skipped 1 previous similar message [ 8515.041746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8515.056140] Lustre: Skipped 1 previous similar message [ 8515.747955] Lustre: server umount lustre-MDT0000 complete [ 8519.490245] LustreError: 286147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859238 with bad export cookie 15536146116064815969 [ 8519.491055] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8519.507656] LustreError: 286147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8519.904982] Lustre: server umount lustre-MDT0001 complete [ 8527.926476] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 05:20:45 (1788859245) [ 8535.347631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8535.931973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8540.461472] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8541.154181] LustreError: 289738:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8541.175772] LustreError: 289738:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8545.256418] LustreError: 289739:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8548.501939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8548.773641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8552.881626] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8560.352892] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8566.302958] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8573.624976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8574.101835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8581.066674] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8588.555581] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8604.128173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8604.144664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8604.145995] LustreError: Skipped 1 previous similar message [ 8604.146960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8604.192068] Lustre: Skipped 1 previous similar message [ 8608.339460] Lustre: server umount lustre-OST0000 complete [ 8614.368962] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8614.372429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8614.375747] Lustre: Skipped 1 previous similar message [ 8614.382842] Lustre: Skipped 1 previous similar message [ 8618.343119] Lustre: server umount lustre-MDT0000 complete [ 8619.497012] LustreError: 289739:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8619.513194] LustreError: 289739:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8622.730190] LustreError: 289720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859341 with bad export cookie 15536146116064816851 [ 8622.733889] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8622.739244] LustreError: 289720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8623.110285] Lustre: server umount lustre-MDT0001 complete [ 8634.246396] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 8637.384643] Key type lgssc unregistered [ 8637.718884] LNet: 293295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8637.746102] LNetError: 293295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8637.787981] LNet: Removed LNI 192.168.203.116@tcp [ 8638.637253] Key type .llcrypt unregistered [ 8638.638617] Key type ._llcrypt unregistered [ 8663.240066] Key type ._llcrypt registered [ 8663.241643] Key type .llcrypt registered [ 8663.347627] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 8677.908343] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8679.128615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8679.204329] alg: No test for adler32 (adler32-zlib) [ 8680.381638] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 8680.636190] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 8682.383224] Key type lgssc registered [ 8684.029746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8726.895704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8726.920728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8728.209584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8728.246877] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8728.354424] Lustre: lustre-MDT0000: new disk, initializing [ 8728.483587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8728.523464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8732.011279] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8743.146286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8743.326828] Lustre: 297117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8743.394675] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8743.403404] Lustre: Skipped 1 previous similar message [ 8743.518648] Lustre: lustre-MDT0001: new disk, initializing [ 8743.632126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8743.692505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8743.726503] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8748.998778] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8759.183609] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8766.137781] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8774.058920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8774.450374] Lustre: lustre-OST2710: new disk, initializing [ 8774.458254] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8774.464866] Lustre: 299084:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8774.623393] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8782.977635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8782.994214] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8783.226587] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8783.415973] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8793.655718] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8801.170237] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8801.412690] Lustre: lustre-OST03e8: new disk, initializing [ 8801.416955] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8801.424072] Lustre: 300149:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8801.522117] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8809.467532] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8810.604037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8810.612814] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8810.721091] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8817.344041] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8827.021501] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8827.300884] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8832.417832] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8832.593259] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8837.841484] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8838.046444] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8843.421815] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8843.845337] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8851.424647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8851.439922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8851.472651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8851.948146] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8851.987530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8852.002554] Lustre: Skipped 2 previous similar messages [ 8857.257908] Lustre: server umount lustre-MDT0000 complete [ 8862.180531] LustreError: 298836:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8862.226860] LustreError: 298836:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 8862.681337] LustreError: 301890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859581 with bad export cookie 7997499684481593468 [ 8862.697574] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8863.107372] Lustre: server umount lustre-MDT0001 complete [ 8876.594844] Lustre: server umount lustre-OST2710 complete [ 8889.660849] Lustre: server umount lustre-OST03e8 complete [ 8909.640854] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 8919.435229] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8955.224713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8955.502187] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8955.534776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8955.675918] Lustre: lustre-MDT0000: new disk, initializing [ 8955.717870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8955.728096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8961.700199] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8973.825965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8973.935540] Lustre: 305401:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8973.997818] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8974.000769] Lustre: Skipped 1 previous similar message [ 8974.111253] Lustre: lustre-MDT0001: new disk, initializing [ 8974.195267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8974.234125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8974.250129] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8979.761834] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8988.102168] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8993.214694] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8999.827930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9000.108567] Lustre: lustre-OST0000: new disk, initializing [ 9000.112861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9000.119481] Lustre: 307373:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9000.225228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9006.785761] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9008.747106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9008.764730] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9008.939340] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9017.559541] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9023.383839] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9023.679624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9028.694530] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9029.013113] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9034.723260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9034.723667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9034.746855] LustreError: Skipped 1 previous similar message [ 9034.750181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9034.781325] Lustre: Skipped 3 previous similar messages [ 9034.796037] Lustre: Skipped 1 previous similar message [ 9037.426622] Lustre: server umount lustre-OST0000 complete [ 9044.964488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9044.973045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9044.990498] Lustre: Skipped 1 previous similar message [ 9045.008900] Lustre: Skipped 1 previous similar message [ 9048.274666] Lustre: server umount lustre-MDT0000 complete [ 9050.080516] LustreError: 306430:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9050.110106] LustreError: 306430:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9052.071525] LustreError: 305394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859770 with bad export cookie 7997499684481595162 [ 9052.079828] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9052.648902] Lustre: server umount lustre-MDT0001 complete [ 9060.999593] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 05:29:38 (1788859778) [ 9062.408277] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9063.859342] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 05:29:41 (1788859781) [ 9067.638917] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9092.118646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9092.387719] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9092.427179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9092.631523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9096.873926] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9106.457646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9106.519130] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9106.561950] Lustre: 311225:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9111.781906] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9133.627776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9133.791260] Lustre: Denying initial registration attempt from nid 192.168.203.116@tcp, specified as failover [ 9133.805154] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9133.810327] LustreError: 312680:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9133.814834] LustreError: 312680:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9133.818658] LustreError: 312680:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9133.863460] Lustre: server umount lustre-OST0000 complete [ 9133.866142] LustreError: 312680:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9142.752338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9142.768129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9142.792885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9147.217442] Lustre: server umount lustre-MDT0000 complete [ 9148.907134] LustreError: 311233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9148.956772] LustreError: 311233:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9152.927816] LustreError: 311219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859871 with bad export cookie 7997499684481596044 [ 9152.947756] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9152.956249] LustreError: 311219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9153.374887] Lustre: server umount lustre-MDT0001 complete [ 9170.281333] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 9172.858970] Key type lgssc unregistered [ 9173.164532] LNet: 313813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9173.180936] LNetError: 313813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9173.213956] LNet: Removed LNI 192.168.203.116@tcp [ 9174.210265] Key type .llcrypt unregistered [ 9174.218889] Key type ._llcrypt unregistered [ 9185.009169] Key type ._llcrypt registered [ 9185.011771] Key type .llcrypt registered [ 9185.100556] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9185.872756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9186.004475] alg: No test for adler32 (adler32-zlib) [ 9187.113443] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 9187.444769] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 9189.175192] Key type lgssc registered [ 9190.588324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9240.374427] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9253.924620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9253.965649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9255.530650] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9255.584631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9255.832277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9260.694717] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9269.162696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9269.344946] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9269.452445] Lustre: 316579:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9269.750070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9274.880538] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9281.993826] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9288.683884] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9308.392208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9308.657074] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9308.895089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9315.264300] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9322.412485] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9326.060715] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9326.075276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9326.090550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9326.560261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9330.760400] Lustre: server umount lustre-OST0000 complete [ 9336.288571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9336.307072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9336.337412] Lustre: Skipped 1 previous similar message [ 9336.346390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9336.361548] Lustre: Skipped 1 previous similar message [ 9340.456167] Lustre: server umount lustre-MDT0000 complete [ 9343.959430] LustreError: 316569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860062 with bad export cookie 11190485015638579218 [ 9343.963211] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9343.970600] LustreError: 316569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9344.351732] Lustre: server umount lustre-MDT0001 complete [ 9354.588979] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 9357.184945] Key type lgssc unregistered [ 9357.489975] LNet: 320150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9357.499147] LNetError: 320150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9357.515279] LNet: Removed LNI 192.168.203.116@tcp [ 9358.465469] Key type .llcrypt unregistered [ 9358.467388] Key type ._llcrypt unregistered [ 9382.748427] Key type ._llcrypt registered [ 9382.750762] Key type .llcrypt registered [ 9382.841221] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9383.694311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9383.795767] alg: No test for adler32 (adler32-zlib) [ 9384.800734] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 9385.011186] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 9386.711226] Key type lgssc registered [ 9387.681708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9394.347078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9394.372600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9395.845979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9399.942839] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9405.414734] LustreError: 321564:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9405.439672] LustreError: 321564:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9408.009216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9408.479989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9413.230560] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9420.686036] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9426.480918] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9435.302967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9435.825775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9443.032357] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9451.039543] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9463.779439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9463.784596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9463.812731] Lustre: Skipped 2 previous similar messages [ 9463.833707] Lustre: Skipped 2 previous similar messages [ 9466.795561] Lustre: server umount lustre-MDT0000 complete [ 9468.899685] LustreError: 321565:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9468.927356] LustreError: 321565:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9471.010945] LustreError: 322298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860189 with bad export cookie 1323992628721622999 [ 9471.021661] LustreError: 322298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9471.023550] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9474.016352] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9474.017963] LustreError: 321560:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9474.030756] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9474.030762] Lustre: Skipped 1 previous similar message [ 9474.070259] LustreError: 321560:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9476.065440] LustreError: 321560:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9476.065531] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9476.092373] LustreError: 321560:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9476.119864] Lustre: Skipped 1 previous similar message [ 9477.868518] Lustre: server umount lustre-MDT0001 complete [ 9484.210584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9494.711547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9495.852486] LustreError: 323397:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9495.959282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9496.037327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9501.715729] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9505.187734] LustreError: 325151:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9505.191384] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9505.218905] LustreError: 325151:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9510.921239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9511.471952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9515.719275] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9516.529512] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9522.926046] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9524.624419] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9530.084168] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9531.774757] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9536.998327] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9537.000026] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9537.007038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9537.028502] Lustre: Skipped 1 previous similar message [ 9537.041750] Lustre: Skipped 1 previous similar message [ 9542.115646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9542.130884] Lustre: Skipped 1 previous similar message [ 9543.021019] Lustre: server umount lustre-OST0000 complete [ 9547.233552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9552.355358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9552.364463] Lustre: Skipped 3 previous similar messages [ 9553.397528] Lustre: server umount lustre-MDT0000 complete [ 9557.418834] LustreError: 321544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860276 with bad export cookie 1323992628721630139 [ 9557.420968] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9557.434035] LustreError: 321544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9558.011772] Lustre: server umount lustre-MDT0001 complete [ 9570.759291] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 9573.871202] Key type lgssc unregistered [ 9574.167631] LNet: 327824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9574.180730] LNetError: 327824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9574.219781] LNet: Removed LNI 192.168.203.116@tcp [ 9575.377984] Key type .llcrypt unregistered [ 9575.391518] Key type ._llcrypt unregistered [ 9600.987095] Key type ._llcrypt registered [ 9600.988893] Key type .llcrypt registered [ 9601.088440] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9602.067911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9602.096960] alg: No test for adler32 (adler32-zlib) [ 9603.281782] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 9603.524034] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 9605.191219] Key type lgssc registered [ 9606.232289] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9613.790469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9613.832154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9615.545038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9619.888477] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9626.091315] LustreError: 329242:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9626.133142] LustreError: 329242:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9629.110109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9629.516877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9634.680969] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9644.031311] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9652.042043] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9659.367582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9659.379293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9659.386433] Lustre: Skipped 1 previous similar message [ 9662.024420] Lustre: server umount lustre-MDT0000 complete [ 9664.480377] LustreError: 329242:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9664.526228] LustreError: 329242:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9666.288403] LustreError: 329222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860385 with bad export cookie 17832814697813974627 [ 9666.309904] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9666.313237] LustreError: 329222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9666.774691] Lustre: server umount lustre-MDT0001 complete [ 9684.568610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9685.037884] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9685.097288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9685.565197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9691.863630] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9702.570911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9702.780065] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9702.872465] Lustre: 331830:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9703.391655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9709.543823] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9718.124426] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9723.250451] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9733.166770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9733.418136] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9733.439249] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9733.453919] LustreError: 333716:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 9733.462325] LustreError: 333716:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9733.466944] LustreError: 333716:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9733.546931] Lustre: server umount lustre-OST0000 complete [ 9733.552702] LustreError: 333716:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9742.896933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9743.088857] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9743.269650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9748.337032] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9752.184336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9754.624390] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9761.132035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9761.217859] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9761.303396] Lustre: lustre-OST0001: new disk, initializing [ 9761.307345] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9761.317499] Lustre: 335258:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9761.397527] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9766.964856] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9770.723299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9770.749415] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9770.881110] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9774.362711] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9781.230128] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9781.233813] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9781.242282] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9781.263644] Lustre: Skipped 2 previous similar messages [ 9783.372983] Lustre: server umount lustre-OST0001 complete [ 9786.353865] LustreError: 335267:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9786.389155] LustreError: 335267:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9787.872277] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9787.881783] LustreError: Skipped 1 previous similar message [ 9787.885011] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9787.894378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9787.899408] Lustre: Skipped 1 previous similar message [ 9791.460236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9791.464465] LustreError: 334182:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9791.475721] Lustre: Skipped 1 previous similar message [ 9793.182342] Lustre: server umount lustre-OST0000 complete [ 9800.672528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9800.684222] LustreError: Skipped 1 previous similar message [ 9800.688188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9800.701334] Lustre: Skipped 1 previous similar message [ 9800.706187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9803.194424] Lustre: server umount lustre-MDT0000 complete [ 9806.839759] LustreError: 331838:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9806.879811] LustreError: 331838:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9807.276686] LustreError: 331823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860526 with bad export cookie 17832814697813975187 [ 9807.291030] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9807.294767] LustreError: 331823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9807.870706] Lustre: server umount lustre-MDT0001 complete [ 9821.132102] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 9825.014689] Key type lgssc unregistered [ 9825.368637] LNet: 337214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9825.377284] LNetError: 337214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9825.401161] LNet: Removed LNI 192.168.203.116@tcp [ 9826.467705] Key type .llcrypt unregistered [ 9826.470817] Key type ._llcrypt unregistered [ 9897.298486] Key type ._llcrypt registered [ 9897.301374] Key type .llcrypt registered [ 9897.407382] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 9911.970023] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9913.428871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9913.479737] alg: No test for adler32 (adler32-zlib) [ 9914.525767] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 9914.719412] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 9916.407226] Key type lgssc registered [ 9917.279604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9951.027773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9951.085122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9952.279996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9952.358723] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9952.485442] Lustre: lustre-MDT0000: new disk, initializing [ 9952.571676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9952.603374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9957.427787] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9967.046682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9967.166260] Lustre: 342147:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9967.212609] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9967.215995] Lustre: Skipped 1 previous similar message [ 9967.282748] Lustre: lustre-MDT0001: new disk, initializing [ 9967.397690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9967.435048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9967.450747] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9971.405687] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9980.597232] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9985.731644] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9991.732839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9992.053513] Lustre: lustre-OST0000: new disk, initializing [ 9992.060328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9992.069213] Lustre: 344113:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9992.211540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9997.989199] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10002.032519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10002.047040] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10002.133370] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10006.998111] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10011.591430] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10011.796849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10016.518961] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10016.680528] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10022.880563] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10022.889750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10022.901223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10022.910164] Lustre: Skipped 1 previous similar message [10022.917278] Lustre: Skipped 1 previous similar message [10024.664539] Lustre: server umount lustre-OST0000 complete [10028.002059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10028.004084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10028.025898] Lustre: Skipped 1 previous similar message [10028.048518] Lustre: Skipped 1 previous similar message [10033.122267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10033.132339] Lustre: Skipped 1 previous similar message [10038.242161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10038.246699] Lustre: Skipped 1 previous similar message [10042.335222] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10042.344537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10042.366204] Lustre: Skipped 1 previous similar message [10042.510303] Lustre: server umount lustre-MDT0000 complete [10046.114229] LustreError: 342140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860764 with bad export cookie 8615794142205382154 [10046.120266] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10046.127745] LustreError: 342140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10046.458576] Lustre: server umount lustre-MDT0001 complete [10056.922489] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 05:46:13 (1788860773) [10066.458250] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 05:46:23 (1788860783) [10074.364697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10075.134529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10079.532761] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10080.229270] LustreError: 346873:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10080.264668] LustreError: 346873:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10084.322529] LustreError: 346872:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10087.369027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10087.804330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10092.151466] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10099.632740] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10104.992504] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10111.286668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10111.657246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10118.171575] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10125.283662] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10134.701917] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10136.760687] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10138.316740] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10140.035511] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10141.926671] Lustre: Failing over lustre-MDT0000 [10142.222974] Lustre: server umount lustre-MDT0000 complete [10143.608173] LustreError: 347606:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10144.227766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10144.242708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10145.761969] LustreError: 346872:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10145.795519] LustreError: 346872:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10150.881568] LustreError: 346873:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10150.905815] LustreError: 346873:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10158.917744] LustreError: 347606:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10158.940558] LustreError: 347606:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10161.201164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10161.401675] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10161.725175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10161.792304] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10165.064838] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10166.777571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10166.830115] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10166.893429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10167.863333] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10178.826222] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10180.592499] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10184.151750] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10185.738359] Lustre: Failing over lustre-MDT0000 [10186.059187] Lustre: server umount lustre-MDT0000 complete [10187.236029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10187.236455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10187.241283] LustreError: 347606:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10187.241293] LustreError: 347606:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10187.250948] Lustre: Skipped 3 previous similar messages [10203.617060] Lustre: 340368:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788860906/real 1788860906] req@0000000055d06d4b x1875756327196928/t0(0) o400->MGC192.168.203.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788860922 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10203.656856] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10204.751993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10214.229689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10214.297217] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10216.259567] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10219.513459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10219.526320] Lustre: Skipped 2 previous similar messages [10219.589156] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10219.679196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10220.731548] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10231.001701] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10232.481504] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10239.973067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10239.977074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10239.991060] Lustre: Skipped 3 previous similar messages [10239.999977] Lustre: Skipped 1 previous similar message [10246.241542] Lustre: server umount lustre-MDT0000 complete [10250.208641] LustreError: 346872:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10250.257301] LustreError: 346872:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 27 previous similar messages [10250.532869] LustreError: 346852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860969 with bad export cookie 8615794142205384898 [10250.556742] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10250.575454] LustreError: 346852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10250.858866] Lustre: server umount lustre-MDT0001 complete [10265.431949] Lustre: server umount lustre-OST0000 complete [10283.154385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10284.116297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10289.484558] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10300.864428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10308.189378] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10318.282684] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10324.342722] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10331.294229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10337.925462] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10344.626430] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10353.276841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10362.848727] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10362.878260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10362.909462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10362.917282] Lustre: Skipped 4 previous similar messages [10368.454830] Lustre: server umount lustre-OST0000 complete [10373.088180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10373.112946] LustreError: Skipped 1 previous similar message [10378.927813] Lustre: server umount lustre-MDT0000 complete [10379.233872] LustreError: 353211:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10379.255160] LustreError: 353211:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [10383.030731] LustreError: 353195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788861101 with bad export cookie 8615794142205385605 [10383.035719] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10383.053538] LustreError: 353195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10383.365451] Lustre: server umount lustre-MDT0001 complete [10398.615418] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [10404.265504] Key type lgssc unregistered [10404.945977] LNet: 356772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10404.982423] LNetError: 356772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10405.041114] LNet: Removed LNI 192.168.203.116@tcp [10406.673852] Key type .llcrypt unregistered [10406.679363] Key type ._llcrypt unregistered [10438.092479] Key type ._llcrypt registered [10438.095391] Key type .llcrypt registered [10438.198957] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [10439.201449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10439.246944] alg: No test for adler32 (adler32-zlib) [10440.374726] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [10440.593351] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [10442.288205] Key type lgssc registered [10443.393832] Lustre: Echo OBD driver; http://www.lustre.org/ [10450.927472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10450.944258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10452.449542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10457.025487] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10462.705943] LustreError: 358288:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10462.722766] LustreError: 358288:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10465.196529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10465.575624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10469.778268] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10477.052717] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10482.377665] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10488.383464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10488.784760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10494.533376] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10498.060691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10502.188413] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10513.401820] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10513.420588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10513.423741] Lustre: Skipped 1 previous similar message [10513.439381] Lustre: Skipped 2 previous similar messages [10515.459853] Lustre: server umount lustre-MDT0000 complete [10518.505574] LustreError: 358283:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10518.537125] LustreError: 358283:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10519.123788] LustreError: 358269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788861237 with bad export cookie 11523061783116342391 [10519.127050] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10519.142299] LustreError: 358269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10519.535039] Lustre: server umount lustre-MDT0001 complete [10533.936228] Lustre: server umount lustre-OST0000 complete [10548.505586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10549.253773] LustreError: 362001:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10549.330243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10553.654081] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10554.347397] LustreError: 362002:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10559.469232] LustreError: 362001:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10562.318374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10562.749339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10567.059854] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10574.855782] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10580.762454] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10587.611734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10587.947771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10594.131823] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10596.174054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10601.518460] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10614.239985] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10614.247240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10614.255332] Lustre: Skipped 1 previous similar message [10614.258674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10617.832912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10617.834207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10621.548850] Lustre: server umount lustre-MDT0000 complete [10622.946982] LustreError: 361998:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10622.973224] LustreError: 361998:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10624.817922] LustreError: 361981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788861343 with bad export cookie 11523061783116343497 [10624.823783] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10624.831322] LustreError: 361981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10625.092319] Lustre: server umount lustre-MDT0001 complete [10640.067190] Lustre: server umount lustre-OST0000 complete [10654.492946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10654.950495] LustreError: 365913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10655.041302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10659.420944] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10670.000996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10675.574722] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10683.524378] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10688.318363] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10694.356620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10694.726996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10694.735182] Lustre: Skipped 1 previous similar message [10699.831346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10700.784804] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10708.817405] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10715.127436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10715.135264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10715.165532] Lustre: Skipped 1 previous similar message [10715.168055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10715.197576] Lustre: Skipped 3 previous similar messages [10720.230758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10720.241651] Lustre: Skipped 1 previous similar message [10720.796764] Lustre: server umount lustre-OST0000 complete [10725.344459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10725.367380] Lustre: Skipped 1 previous similar message [10730.464804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10730.471315] Lustre: Skipped 3 previous similar messages [10731.366704] Lustre: server umount lustre-MDT0000 complete [10735.070847] LustreError: 368697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788861453 with bad export cookie 11523061783116344526 [10735.083407] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10735.091207] LustreError: 368697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10735.333369] Lustre: server umount lustre-MDT0001 complete [10745.121122] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [10747.846490] Key type lgssc unregistered [10748.103782] LNet: 369449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10748.111905] LNetError: 369449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10748.131731] LNet: Removed LNI 192.168.203.116@tcp [10748.945647] Key type .llcrypt unregistered [10748.948630] Key type ._llcrypt unregistered [10775.891205] Key type ._llcrypt registered [10775.893874] Key type .llcrypt registered [10776.045707] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [10777.721494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10777.754977] alg: No test for adler32 (adler32-zlib) [10778.864249] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [10779.051925] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [10780.767196] Key type lgssc registered [10781.756591] Lustre: Echo OBD driver; http://www.lustre.org/ [10789.472556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10789.519245] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10789.536619] LustreError: 370886:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10789.572119] LustreError: 370886:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10789.587186] LustreError: 370886:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10789.607863] LustreError: 370886:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10789.619735] LustreError: 370886:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10797.312854] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10797.323753] LustreError: 371283:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10797.366319] LustreError: 371283:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10797.381243] LustreError: 371283:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10797.390661] LustreError: 371283:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10797.403483] LustreError: 371283:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10816.676269] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [10819.546907] Key type lgssc unregistered [10819.840779] LNet: 372287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10819.860397] LNetError: 372287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10819.885777] LNet: Removed LNI 192.168.203.116@tcp [10820.863760] Key type .llcrypt unregistered [10820.867342] Key type ._llcrypt unregistered [10840.081461] Key type ._llcrypt registered [10840.083921] Key type .llcrypt registered [10840.189114] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [10858.485995] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [10859.512119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10859.549651] alg: No test for adler32 (adler32-zlib) [10860.741455] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [10860.996570] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [10862.647619] Key type lgssc registered [10863.596791] Lustre: Echo OBD driver; http://www.lustre.org/ [10894.454598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10894.486625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10895.687150] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10895.714991] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10895.771234] Lustre: lustre-MDT0000: new disk, initializing [10895.847382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10895.864974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10899.581685] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10909.352821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10909.501882] Lustre: 375439:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10909.587046] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10909.591499] Lustre: Skipped 1 previous similar message [10909.685641] Lustre: lustre-MDT0001: new disk, initializing [10909.781499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10909.829223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10909.848416] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10914.526671] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10923.341659] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10928.479146] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10935.392766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10935.655924] Lustre: lustre-OST0000: new disk, initializing [10935.660263] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10935.669990] Lustre: 377406:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10935.774292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10942.646120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10942.663750] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10942.669527] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10942.922285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10953.283775] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10959.807176] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10960.188882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10965.348798] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10965.572769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10968.549981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10968.556208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10968.571227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10968.585803] Lustre: Skipped 1 previous similar message [10973.571355] Lustre: server umount lustre-OST0000 complete [10978.784942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10978.789971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10978.804193] Lustre: Skipped 2 previous similar messages [10978.822793] Lustre: Skipped 1 previous similar message [10983.325692] Lustre: server umount lustre-MDT0000 complete [10986.868594] LustreError: 375432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788861705 with bad export cookie 2868867693773749730 [10986.890731] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10987.140443] Lustre: server umount lustre-MDT0001 complete [10994.838931] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10996.472328] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 06:01:53 (1788861713) [11002.975840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11003.325334] LustreError: 379878:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11003.337846] LustreError: 379878:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11003.386889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11008.486266] LustreError: 379879:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11009.257242] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11013.603855] LustreError: 379878:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11018.293662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11018.707655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11023.578936] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11031.025869] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11037.611811] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11043.754739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11044.105643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11050.126852] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11058.138752] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11065.131695] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11065.376487] Lustre: lustre-OST0001: new disk, initializing [11065.386724] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11065.394545] Lustre: 382714:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11065.542514] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11071.627722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11071.643502] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11071.761414] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11072.529352] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11082.291850] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11087.332077] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11087.338836] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11087.345360] LustreError: Skipped 2 previous similar messages [11087.348551] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11087.389583] Lustre: Skipped 1 previous similar message [11090.292660] Lustre: lustre-OST0001: Not available for connect from 192.168.203.16@tcp (stopping) [11090.304137] Lustre: Skipped 1 previous similar message [11093.263619] Lustre: server umount lustre-OST0001 complete [11095.373667] LustreError: 381699:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11102.691396] LustreError: 381697:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11102.721415] LustreError: 381697:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11112.928905] LustreError: 381697:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11112.930682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11112.938068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11112.938076] Lustre: Skipped 2 previous similar messages [11112.948672] LustreError: 381697:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11112.997801] Lustre: Skipped 1 previous similar message [11123.168927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11123.188954] Lustre: Skipped 4 previous similar messages [11125.215220] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11125.575503] Lustre: server umount lustre-OST0000 complete [11131.363765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11131.375520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11136.058302] Lustre: server umount lustre-MDT0000 complete [11138.528610] LustreError: 379874:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11138.550106] LustreError: 379874:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11139.727887] LustreError: 379860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788861858 with bad export cookie 2868867693773750612 [11139.741982] LustreError: 379860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11139.742856] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11146.310775] Lustre: server umount lustre-MDT0001 complete [11156.472609] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [11158.978987] Key type lgssc unregistered [11159.175462] LNet: 384703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11159.184169] LNetError: 384703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11159.201804] LNet: Removed LNI 192.168.203.116@tcp [11160.108218] Key type .llcrypt unregistered [11160.111086] Key type ._llcrypt unregistered [11208.230888] Key type ._llcrypt registered [11208.233305] Key type .llcrypt registered [11208.346791] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [11209.285378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11209.301663] alg: No test for adler32 (adler32-zlib) [11210.452078] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [11210.632932] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [11212.359196] Key type lgssc registered [11213.427338] Lustre: Echo OBD driver; http://www.lustre.org/ [11220.156302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11220.188316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11221.487248] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11221.510557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11221.644493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11226.108427] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11234.253518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11234.341993] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11234.364996] Lustre: 387499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11234.556058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11238.833657] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11245.781757] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11251.539915] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11255.265172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11255.278227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11255.300613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11257.312906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11257.313806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11259.731427] Lustre: server umount lustre-MDT0000 complete [11263.354872] LustreError: 387492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788861982 with bad export cookie 8007229988437114655 [11263.363968] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11263.752310] Lustre: server umount lustre-MDT0001 complete [11272.109195] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 06:06:29 (1788861989) [11278.368300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11278.815528] LustreError: 390011:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11278.827518] LustreError: 390011:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11278.899501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11283.240331] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11283.943520] LustreError: 390012:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11289.068813] LustreError: 390011:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11291.828784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11292.315867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11296.971409] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11304.360381] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11310.013709] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11316.166149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11316.310683] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11316.486380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11322.718832] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11329.499395] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11337.400611] Lustre: Permanently deactivating lustre-OST0000 [11337.413369] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11343.926205] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11344.883621] LustreError: 392949:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11344.891684] LustreError: 392949:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11344.896675] LustreError: 392949:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11347.506459] LustreError: 392998:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11347.510867] LustreError: 392998:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11355.316800] Lustre: server umount lustre-OST0000 complete [11357.592412] LustreError: 393247:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11357.597175] LustreError: 393247:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11363.309182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11363.313936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11363.327787] Lustre: Skipped 1 previous similar message [11363.340416] Lustre: Skipped 2 previous similar messages [11368.417641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11368.436032] Lustre: Skipped 1 previous similar message [11369.541317] Lustre: server umount lustre-MDT0000 complete [11372.820692] LustreError: 389992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788862091 with bad export cookie 8007229988437115236 [11372.823773] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11372.834822] LustreError: 389992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11382.401504] Lustre: server umount lustre-MDT0001 complete [11390.371172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11394.477800] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11396.604640] LustreError: 394408:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11402.274740] Lustre: 394746:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.116@tcp [11403.941646] Lustre: server umount MGS complete [11415.096251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11415.576527] LustreError: 395470:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11415.604784] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11415.610161] Lustre: Skipped 1 previous similar message [11415.670735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11420.288152] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11421.161292] LustreError: 395471:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11430.107021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11430.378217] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11435.605793] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11444.362779] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11450.559591] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11453.012627] Lustre: Permanently reactivating lustre-OST0000 [11453.035684] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11453.043131] Lustre: Skipped 2 previous similar messages [11479.007352] LustreError: 397204:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11486.314225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11486.647977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11486.652761] Lustre: Skipped 1 previous similar message [11492.218061] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11498.807333] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11504.100360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11504.103964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11504.112139] LustreError: Skipped 1 previous similar message [11504.131513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11504.146100] Lustre: Skipped 1 previous similar message [11509.004862] Lustre: server umount lustre-OST0000 complete [11514.342335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11514.363199] Lustre: Skipped 1 previous similar message [11514.373504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11514.376611] Lustre: Skipped 1 previous similar message [11519.991836] Lustre: server umount lustre-MDT0000 complete [11524.057415] LustreError: 395452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788862242 with bad export cookie 8007229988437116286 [11524.064972] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11524.081043] LustreError: 395452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11524.439359] Lustre: server umount lustre-MDT0001 complete [11533.289902] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [11535.655949] Key type lgssc unregistered [11535.919156] LNet: 399884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11535.935948] LNetError: 399884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11535.962835] LNet: Removed LNI 192.168.203.116@tcp [11536.763698] Key type .llcrypt unregistered [11536.765896] Key type ._llcrypt unregistered [11552.283183] Key type ._llcrypt registered [11552.289188] Key type .llcrypt registered [11552.396621] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [11565.307293] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [11566.432077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11566.456448] alg: No test for adler32 (adler32-zlib) [11567.449354] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [11567.641960] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [11569.303208] Key type lgssc registered [11570.436789] Lustre: Echo OBD driver; http://www.lustre.org/ [11601.975769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11602.011861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11603.284108] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11603.307103] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11603.375838] Lustre: lustre-MDT0000: new disk, initializing [11603.443199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11603.463960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11607.176913] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11615.858147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11615.959479] Lustre: 403013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11616.024263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11616.028741] Lustre: Skipped 1 previous similar message [11616.106616] Lustre: lustre-MDT0001: new disk, initializing [11616.182921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11616.215860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11616.223304] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11619.379887] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11627.159404] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11631.946581] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11637.430114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11637.647995] Lustre: lustre-OST0000: new disk, initializing [11637.651672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11637.657700] Lustre: 404982:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11637.753086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11642.828742] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11644.990492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11644.998780] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11645.104146] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11651.380288] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11655.348439] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11655.553330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11659.499163] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11659.710399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11665.898537] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11665.912652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11665.943072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11665.951697] Lustre: Skipped 1 previous similar message [11667.536109] Lustre: server umount lustre-OST0000 complete [11671.010545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11671.035420] Lustre: Skipped 1 previous similar message [11671.040900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11676.128955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11676.141250] Lustre: Skipped 2 previous similar messages [11676.748022] Lustre: server umount lustre-MDT0000 complete [11679.911839] LustreError: 403008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788862398 with bad export cookie 7542141065431122969 [11679.923349] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11680.255915] Lustre: server umount lustre-MDT0001 complete [11686.461369] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11687.636955] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 06:13:25 (1788862405) [11692.912063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11693.411326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11696.706155] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11698.657389] LustreError: 407442:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11698.679373] LustreError: 407442:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11703.592620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11703.777240] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11703.912058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11707.640700] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11713.954626] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11719.168942] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11724.754498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11725.078244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11730.809953] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11737.368893] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11738.789388] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11741.059427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11748.841663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11748.851227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11748.862692] Lustre: Skipped 1 previous similar message [11748.864839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11748.900538] LustreError: Skipped 1 previous similar message [11751.575797] Lustre: server umount lustre-OST0000 complete [11754.985863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11754.993908] LustreError: Skipped 1 previous similar message [11755.001407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11755.008824] Lustre: Skipped 1 previous similar message [11759.072046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11759.079376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11759.079724] Lustre: Skipped 2 previous similar messages [11769.311138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11769.454431] Lustre: server umount lustre-MDT0000 complete [11772.672240] LustreError: 407421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788862491 with bad export cookie 7542141065431123844 [11772.677333] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11772.687439] LustreError: 407421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11773.066387] Lustre: server umount lustre-MDT0001 complete [11782.120267] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [11784.660335] Key type lgssc unregistered [11784.898613] LNet: 411164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11784.906779] LNetError: 411164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11784.917410] LNet: Removed LNI 192.168.203.116@tcp [11785.800570] Key type .llcrypt unregistered [11785.803537] Key type ._llcrypt unregistered [11806.414324] Key type ._llcrypt registered [11806.416690] Key type .llcrypt registered [11806.560440] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [11806.913301] Key type .llcrypt unregistered [11806.916785] Key type ._llcrypt unregistered [11823.184613] Key type ._llcrypt registered [11823.190852] Key type .llcrypt registered [11823.284973] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [11824.137036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11824.196590] alg: No test for adler32 (adler32-zlib) [11825.286145] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [11825.482891] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [11827.151955] Key type lgssc registered [11828.181406] Lustre: Echo OBD driver; http://www.lustre.org/ [11835.815476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11835.832728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11837.249216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11841.130325] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11847.649831] LustreError: 413372:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11847.664832] LustreError: 413372:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11848.383161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11848.623318] LustreError: 413371:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11848.644913] LustreError: 413371:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11848.655056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11853.792242] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11853.793285] LustreError: 413372:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11853.821243] LustreError: 413372:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11858.914598] LustreError: 413371:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11858.943643] LustreError: 413371:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11860.094214] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11863.024492] LustreError: 413367:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11863.047892] LustreError: 413367:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11863.054434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11865.872269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11866.158767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11869.250500] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11881.452029] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11881.472857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11881.474054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11885.890385] Lustre: server umount lustre-OST0000 complete [11891.682797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11891.683497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11891.703657] Lustre: Skipped 1 previous similar message [11891.707523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11891.727947] Lustre: Skipped 2 previous similar messages [11895.694463] Lustre: server umount lustre-MDT0000 complete [11896.805668] LustreError: 413372:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11896.826560] LustreError: 413372:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11898.751248] LustreError: 413353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788862617 with bad export cookie 5849943685925429283 [11898.757952] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11898.763755] LustreError: 413353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11899.211333] Lustre: server umount lustre-MDT0001 complete [11908.435854] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [11910.878931] Key type lgssc unregistered [11911.158581] LNet: 416517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11911.166799] LNetError: 416517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11911.183647] LNet: Removed LNI 192.168.203.116@tcp [11911.928166] Key type .llcrypt unregistered [11911.930930] Key type ._llcrypt unregistered [11931.569046] Key type ._llcrypt registered [11931.573167] Key type .llcrypt registered [11931.673775] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [11932.689751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11932.712651] alg: No test for adler32 (adler32-zlib) [11933.800798] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [11933.997508] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [11935.655281] Key type lgssc registered [11936.669924] Lustre: Echo OBD driver; http://www.lustre.org/ [11942.882357] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11942.894201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11958.431424] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11963.553132] LustreError: 417882:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.116@tcp: failed processing log, type 4: rc = -110 [11990.175195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11995.486551] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12002.285438] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12008.160937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12008.706794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12008.775737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12012.912464] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12014.051492] LustreError: 417908:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12014.062750] LustreError: 417908:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12019.181699] LustreError: 417907:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12020.846852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12021.218910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12025.508523] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12032.641411] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12037.857554] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12046.818905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12046.827255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12046.842762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12047.743628] Lustre: server umount lustre-OST0000 complete [12051.935960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12051.954370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12051.979344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12055.014860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12055.023803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12055.083290] Lustre: Skipped 1 previous similar message [12060.129797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12060.147296] Lustre: Skipped 1 previous similar message [12061.335911] Lustre: server umount lustre-MDT0000 complete [12064.289742] LustreError: 418916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12064.315891] LustreError: 418916:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12065.167081] LustreError: 420697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788862784 with bad export cookie 13872662291809361673 [12065.174313] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12065.187653] LustreError: 420697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12065.524114] Lustre: server umount lustre-MDT0001 complete [12076.493549] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12079.150297] Key type lgssc unregistered [12079.448932] LNet: 421449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12079.456474] LNetError: 421449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12079.488491] LNet: Removed LNI 192.168.203.116@tcp [12080.200511] Key type .llcrypt unregistered [12080.204138] Key type ._llcrypt unregistered [12105.711849] Key type ._llcrypt registered [12105.722320] Key type .llcrypt registered [12105.846694] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [12106.742322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12106.891378] alg: No test for adler32 (adler32-zlib) [12107.941995] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [12108.177652] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [12109.855259] Key type lgssc registered [12111.130622] Lustre: Echo OBD driver; http://www.lustre.org/ [12118.246502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12118.275869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12119.761321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12123.483483] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12129.261388] LustreError: 422864:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12129.292560] LustreError: 422864:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12131.094896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12131.324666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12135.467899] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12142.533518] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12147.564523] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12153.687541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12154.108801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12159.493851] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12164.352679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12166.328211] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12169.702768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12169.709581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12169.713673] Lustre: Skipped 2 previous similar messages [12169.721647] Lustre: Skipped 1 previous similar message [12173.150230] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.16@tcp (stopping) [12174.816863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12174.828243] Lustre: Skipped 2 previous similar messages [12176.156908] Lustre: server umount lustre-MDT0000 complete [12178.373268] LustreError: 422843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.16@tcp arrived at 1788862897 with bad export cookie 4604589799620671433 [12178.394984] LustreError: 422843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12179.956109] LustreError: 422860:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12179.975985] LustreError: 422860:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12180.042851] LustreError: 422844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788862898 with bad export cookie 4604589799620670537 [12180.061714] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12183.008532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12183.017560] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12183.046318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12185.056624] LustreError: 422864:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12185.072176] LustreError: 422864:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12186.430290] Lustre: server umount lustre-OST0000 complete [12189.159658] LustreError: 423893:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12189.184843] LustreError: 423893:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12192.837938] Lustre: server umount lustre-MDT0001 complete [12202.653687] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12205.470373] Key type lgssc unregistered [12205.796449] LNet: 426547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12205.802364] LNetError: 426547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12205.818357] LNet: Removed LNI 192.168.203.116@tcp [12206.533868] Key type .llcrypt unregistered [12206.537208] Key type ._llcrypt unregistered [12228.666611] Key type ._llcrypt registered [12228.668598] Key type .llcrypt registered [12228.746894] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [12229.862553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12229.983283] alg: No test for adler32 (adler32-zlib) [12231.056218] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [12231.294875] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [12232.935238] Key type lgssc registered [12233.959558] Lustre: Echo OBD driver; http://www.lustre.org/ [12241.096864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12241.123827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12242.615966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12246.442197] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12253.158920] LustreError: 427958:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12253.171920] LustreError: 427958:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12253.559062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12253.915388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12257.993369] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12264.768870] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12270.462900] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12276.900072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12277.298410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12283.080336] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12288.562419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12290.395513] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12295.136841] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12295.147068] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12295.166533] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12298.088334] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (stopping) [12298.728530] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12301.184671] Lustre: server umount lustre-MDT0001 complete [12303.842634] LustreError: 427958:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12303.855676] LustreError: 427958:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12308.960888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12308.964398] LustreError: 427959:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12308.977654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12308.999195] Lustre: Skipped 2 previous similar messages [12313.291614] Lustre: server umount lustre-OST0000 complete [12314.083383] LustreError: 427958:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12317.083296] Lustre: server umount lustre-MDT0000 complete [12329.342475] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12332.134513] Key type lgssc unregistered [12332.470836] LNet: 431639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12332.482035] LNetError: 431639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12332.500190] LNet: Removed LNI 192.168.203.116@tcp [12333.275523] Key type .llcrypt unregistered [12333.278365] Key type ._llcrypt unregistered [12358.167799] Key type ._llcrypt registered [12358.171186] Key type .llcrypt registered [12358.295295] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [12372.977620] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [12373.834820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12373.986320] alg: No test for adler32 (adler32-zlib) [12375.055861] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [12375.329502] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [12376.999352] Key type lgssc registered [12377.987309] Lustre: Echo OBD driver; http://www.lustre.org/ [12411.588420] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [12427.209481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12427.245285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12428.517934] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12428.553248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12428.657339] Lustre: lustre-MDT0000: new disk, initializing [12428.738297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12428.762582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12432.755966] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12446.568098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12446.782505] Lustre: lustre-OST0000: new disk, initializing [12446.787469] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12446.790866] Lustre: Skipped 1 previous similar message [12446.794777] Lustre: 436673:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12446.871167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12452.472346] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12455.570444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12455.581066] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12455.617246] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12461.458607] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12474.697612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12474.842557] Lustre: 435657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12474.927615] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12475.013646] Lustre: lustre-MDT0001: new disk, initializing [12475.094373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12475.137228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12475.148425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12479.042151] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12487.875673] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12488.010541] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12491.449818] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12491.649695] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12496.232671] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (stopping) [12496.358191] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12496.370249] Lustre: Skipped 1 previous similar message [12501.329719] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (stopping) [12501.337669] Lustre: Skipped 2 previous similar messages [12501.975424] Lustre: server umount lustre-MDT0001 complete [12511.719146] LustreError: 435668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12511.731455] LustreError: 435668:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12516.833375] LustreError: 436680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12516.872349] LustreError: 436680:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12521.953125] LustreError: 435669:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12521.954575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12522.011795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12522.018152] Lustre: Skipped 2 previous similar messages [12526.407227] Lustre: server umount lustre-OST0000 complete [12527.074649] LustreError: 435668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12536.489854] Lustre: server umount lustre-MDT0000 complete [12549.564848] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12552.290187] Key type lgssc unregistered [12552.714803] LNet: 440420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12552.732932] LNetError: 440420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12552.772664] LNet: Removed LNI 192.168.203.116@tcp [12553.683201] Key type .llcrypt unregistered [12553.685061] Key type ._llcrypt unregistered [12590.714576] Key type ._llcrypt registered [12590.716765] Key type .llcrypt registered [12590.792570] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12591.063277] Key type .llcrypt unregistered [12591.065645] Key type ._llcrypt unregistered [12605.778344] Key type ._llcrypt registered [12605.780226] Key type .llcrypt registered [12605.869228] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [12607.100830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12607.193535] alg: No test for adler32 (adler32-zlib) [12608.244304] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [12608.570098] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [12610.263221] Key type lgssc registered [12611.441697] Lustre: Echo OBD driver; http://www.lustre.org/ [12634.077871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12634.103218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12635.355521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12635.475618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12635.585312] Lustre: lustre-MDT0000: new disk, initializing [12635.667181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12635.683381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12639.102238] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12648.643464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12648.810710] Lustre: 443795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12648.910535] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12648.914344] Lustre: Skipped 1 previous similar message [12649.005606] Lustre: lustre-MDT0001: new disk, initializing [12649.131497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12649.200605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12649.221576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12653.560256] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12664.804983] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12672.112974] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12678.793220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12679.076560] Lustre: lustre-OST0000: new disk, initializing [12679.079909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12679.087762] Lustre: 445766:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12679.184976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12685.414496] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12686.514030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12686.544858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12686.647539] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12694.937200] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12702.184039] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12702.189284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12702.197052] Lustre: Skipped 1 previous similar message [12702.208868] Lustre: Skipped 2 previous similar messages [12706.997608] Lustre: server umount lustre-MDT0000 complete [12707.298171] LustreError: 443802:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12707.318199] LustreError: 443802:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12710.501188] LustreError: 443787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788863429 with bad export cookie 16929282742753105515 [12710.506575] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12710.517319] LustreError: 443787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12710.932723] Lustre: server umount lustre-MDT0001 complete [12723.480408] Lustre: server umount lustre-OST0000 complete [12733.393918] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 06:30:50 (1788863450) [12740.017897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12740.469095] LustreError: 447795:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12740.548514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12744.946683] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12745.702047] LustreError: 447796:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12750.833192] LustreError: 447795:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12752.179662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12752.485172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12756.411565] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12763.053624] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12767.525283] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12786.230237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12786.384305] Lustre: Found index 0 for lustre-OST0000, updating log [12786.605445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12791.611627] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12796.905820] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12811.187355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12819.424682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12819.431922] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12819.440476] Lustre: Skipped 1 previous similar message [12819.448216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12820.517309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12820.526829] Lustre: Skipped 1 previous similar message [12826.611788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12826.620920] Lustre: Skipped 1 previous similar message [12831.201730] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12831.349440] Lustre: server umount lustre-OST0000 complete [12845.152786] LNet: 1 local NIs in recovery (showing 1): 192.168.203.116@tcp [12848.096302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12848.114555] Lustre: Skipped 1 previous similar message [12848.125438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12852.542979] Lustre: server umount lustre-MDT0000 complete [12853.223240] LustreError: 448523:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12853.255077] LustreError: 448523:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12855.853442] LustreError: 449638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788863574 with bad export cookie 16929282742753106908 [12855.863974] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12855.875188] LustreError: 449638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12856.097917] Lustre: server umount lustre-MDT0001 complete [12862.792534] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 06:33:00 (1788863580) [13478.238328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13478.754409] LustreError: 454228:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13478.824931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13483.897751] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [13484.003789] LustreError: 454229:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13489.120679] LustreError: 454228:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13493.281911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13493.620208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13499.275696] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [13508.228861] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13513.800633] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13533.615303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13533.808252] Lustre: Found index 0 for lustre-OST0000, updating log [13534.121858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13537.375387] LNet: 1 local NIs in recovery (showing 1): 192.168.203.116@tcp [13540.020340] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [13547.192958] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13549.195859] Lustre: server umount lustre-OST0000 complete [13553.702287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13553.714319] Lustre: Skipped 1 previous similar message [13553.721509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13553.727990] Lustre: Skipped 2 previous similar messages [13554.785603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13554.795871] Lustre: Skipped 1 previous similar message [13556.835428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13556.837601] Lustre: Skipped 2 previous similar messages [13559.904297] LustreError: 454229:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13559.925902] LustreError: 454229:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13560.991331] Lustre: server umount lustre-MDT0000 complete [13564.155875] LustreError: 454210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788864283 with bad export cookie 16929282742753107993 [13564.167045] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13564.428314] Lustre: server umount lustre-MDT0001 complete [13592.776770] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [13597.791378] LNet: 1 local NIs in recovery (showing 1): 192.168.203.116@tcp [13601.611503] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [13636.948880] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 06:45:54 (1788864354) [13652.214351] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [13654.958615] Key type lgssc unregistered [13655.271332] LNet: 460835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13655.278580] LNetError: 460835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13739.245691] LNet: Removed LNI 192.168.203.116@tcp [13740.335244] Key type .llcrypt unregistered [13740.340831] Key type ._llcrypt unregistered [13758.400253] Key type ._llcrypt registered [13758.402794] Key type .llcrypt registered [13758.494735] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [13759.456664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13759.719335] alg: No test for adler32 (adler32-zlib) [13761.131702] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [13761.534816] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [13763.279235] Key type lgssc registered [13764.524057] Lustre: Echo OBD driver; http://www.lustre.org/ [13768.656188] LNet: Added LNI 192.168.203.116@tcp100 [8/256/0/180] [13769.479800] LNet: Added LNI 192.168.203.116@tcp101 [8/256/0/180] [13770.368597] LNet: Added LNI 192.168.203.116@tcp102 [8/256/0/180] [13771.157128] LNet: Added LNI 192.168.203.116@tcp103 [8/256/0/180] [13771.939674] LNet: Added LNI 192.168.203.116@tcp104 [8/256/0/180] [13772.950785] LNet: Added LNI 192.168.203.116@tcp105 [8/256/0/180] [13773.861299] LNet: Added LNI 192.168.203.116@tcp106 [8/256/0/180] [13774.793609] LNet: Added LNI 192.168.203.116@tcp107 [8/256/0/180] [13775.530156] LNet: Added LNI 192.168.203.116@tcp108 [8/256/0/180] [13776.321502] LNet: Added LNI 192.168.203.116@tcp109 [8/256/0/180] [13777.220250] LNet: Added LNI 192.168.203.116@tcp110 [8/256/0/180] [13778.077563] LNet: Added LNI 192.168.203.116@tcp111 [8/256/0/180] [13778.795452] LNet: Added LNI 192.168.203.116@tcp112 [8/256/0/180] [13779.620927] LNet: Added LNI 192.168.203.116@tcp113 [8/256/0/180] [13780.556648] LNet: Added LNI 192.168.203.116@tcp114 [8/256/0/180] [13781.322677] LNet: Added LNI 192.168.203.116@tcp115 [8/256/0/180] [13782.145089] LNet: Added LNI 192.168.203.116@tcp116 [8/256/0/180] [13783.051653] LNet: Added LNI 192.168.203.116@tcp117 [8/256/0/180] [13783.993680] LNet: Added LNI 192.168.203.116@tcp118 [8/256/0/180] [13784.997355] LNet: Added LNI 192.168.203.116@tcp119 [8/256/0/180] [13785.977467] LNet: Added LNI 192.168.203.116@tcp120 [8/256/0/180] [13786.924833] LNet: Added LNI 192.168.203.116@tcp121 [8/256/0/180] [13787.680556] LNet: Added LNI 192.168.203.116@tcp122 [8/256/0/180] [13788.555897] LNet: Added LNI 192.168.203.116@tcp123 [8/256/0/180] [13789.342808] LNet: Added LNI 192.168.203.116@tcp124 [8/256/0/180] [13790.230662] LNet: Added LNI 192.168.203.116@tcp125 [8/256/0/180] [13791.083777] LNet: Added LNI 192.168.203.116@tcp126 [8/256/0/180] [13792.086580] LNet: Added LNI 192.168.203.116@tcp127 [8/256/0/180] [13792.915530] LNet: Added LNI 192.168.203.116@tcp128 [8/256/0/180] [13793.757645] LNet: Added LNI 192.168.203.116@tcp129 [8/256/0/180] [13794.580070] LNet: Added LNI 192.168.203.116@tcp130 [8/256/0/180] [13795.351444] LNet: Added LNI 192.168.203.116@tcp131 [8/256/0/180] [13796.156897] LNet: Added LNI 192.168.203.116@tcp132 [8/256/0/180] [13797.017895] LNet: Added LNI 192.168.203.116@tcp133 [8/256/0/180] [13797.832578] LNet: Added LNI 192.168.203.116@tcp134 [8/256/0/180] [13798.737994] LNet: Added LNI 192.168.203.116@tcp135 [8/256/0/180] [13804.363706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13804.412383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13805.521428] Lustre: 463863:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.116@tcp: can't add NID 192.168.203.116@tcp130: rc = -75 [13805.644818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13805.667329] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13805.720817] Lustre: lustre-MDT0000: new disk, initializing [13805.782404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13805.798054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13809.291464] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [13818.546983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13818.672386] Lustre: 463878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13818.745675] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13818.752953] Lustre: Skipped 1 previous similar message [13818.818414] Lustre: lustre-MDT0001: new disk, initializing [13818.854978] Lustre: 464680:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.116@tcp: can't add NID 192.168.203.116@tcp130: rc = -75 [13818.862632] Lustre: 464680:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13818.997233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13819.065895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13819.075453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13821.193973] Lustre: 464811:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.116@tcp: can't add NID 192.168.203.116@tcp130: rc = -75 [13821.201284] Lustre: 464811:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13822.999554] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [13831.018651] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13836.039892] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13842.217302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13842.604595] Lustre: lustre-OST0000: new disk, initializing [13842.608489] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13842.639898] Lustre: 465851:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13842.797839] Lustre: 465856:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.116@tcp: can't add NID 192.168.203.116@tcp130: rc = -75 [13842.810769] Lustre: 465856:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13842.888560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13849.299237] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [13850.154720] Lustre: 466228:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.116@tcp: can't add NID 192.168.203.116@tcp130: rc = -75 [13850.172446] Lustre: 466228:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13850.240122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13850.258029] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13850.434286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13860.906254] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13871.073593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13871.075893] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13871.091110] LustreError: Skipped 1 previous similar message [13871.092581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13871.095462] Lustre: Skipped 1 previous similar message [13871.100139] Lustre: Skipped 1 previous similar message [13876.192651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13876.200227] Lustre: Skipped 1 previous similar message [13881.313320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13881.324591] Lustre: Skipped 1 previous similar message [13883.871504] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13884.024887] Lustre: server umount lustre-OST0000 complete [13890.528500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13890.539996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13890.558643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13891.553689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13896.675378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13896.683780] Lustre: Skipped 3 previous similar messages [13901.279633] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13901.523197] Lustre: server umount lustre-MDT0000 complete [13904.540498] LustreError: 463872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788864623 with bad export cookie 13963297995876348143 [13904.544537] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13904.553695] LustreError: 463872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13904.931979] Lustre: server umount lustre-MDT0001 complete [13914.601337] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [13916.878963] Key type lgssc unregistered [13917.176356] LNet: 467770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13917.186118] LNetError: 467770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13917.197429] LNet: Removed LNI 192.168.203.116@tcp [13918.239659] LNet: Removed LNI 192.168.203.116@tcp100 [13918.244175] LNetError: 467770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13918.251133] LNetError: 467770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13918.266115] LNet: Removed LNI 192.168.203.116@tcp101 [13919.327145] LNet: Removed LNI 192.168.203.116@tcp102 [13919.329117] LNetError: 467770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13919.337854] LNetError: 467770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13919.344401] LNet: Removed LNI 192.168.203.116@tcp103 [13920.351763] LNet: Removed LNI 192.168.203.116@tcp104 [13921.375145] LNet: Removed LNI 192.168.203.116@tcp105 [13921.383055] LNetError: 467770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13921.391714] LNetError: 467770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13921.410911] LNet: Removed LNI 192.168.203.116@tcp106 [13922.463357] LNet: Removed LNI 192.168.203.116@tcp107 [13923.487362] LNet: Removed LNI 192.168.203.116@tcp108 [13924.511209] LNet: Removed LNI 192.168.203.116@tcp109 [13925.535416] LNet: Removed LNI 192.168.203.116@tcp110 [13925.544521] LNetError: 467770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13925.556083] LNetError: 467770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13925.560493] LNet: Removed LNI 192.168.203.116@tcp111 [13926.623240] LNet: Removed LNI 192.168.203.116@tcp112 [13927.647303] LNet: Removed LNI 192.168.203.116@tcp113 [13928.674519] LNet: Removed LNI 192.168.203.116@tcp114 [13929.695281] LNet: Removed LNI 192.168.203.116@tcp115 [13929.700412] LNet: Removed LNI 192.168.203.116@tcp116 [13929.706377] LNet: Removed LNI 192.168.203.116@tcp117 [13929.716603] LNet: Removed LNI 192.168.203.116@tcp118 [13929.719588] LNet: Removed LNI 192.168.203.116@tcp119 [13929.723551] LNet: Removed LNI 192.168.203.116@tcp120 [13929.726717] LNet: Removed LNI 192.168.203.116@tcp121 [13929.728712] LNet: Removed LNI 192.168.203.116@tcp122 [13929.731614] LNet: Removed LNI 192.168.203.116@tcp123 [13929.734562] LNet: Removed LNI 192.168.203.116@tcp124 [13929.737770] LNet: Removed LNI 192.168.203.116@tcp125 [13929.740569] LNet: Removed LNI 192.168.203.116@tcp126 [13929.743662] LNet: Removed LNI 192.168.203.116@tcp127 [13929.748621] LNet: Removed LNI 192.168.203.116@tcp128 [13929.754478] LNet: Removed LNI 192.168.203.116@tcp129 [13929.761783] LNet: Removed LNI 192.168.203.116@tcp130 [13929.766948] LNet: Removed LNI 192.168.203.116@tcp131 [13929.775653] LNet: Removed LNI 192.168.203.116@tcp132 [13929.789906] LNet: Removed LNI 192.168.203.116@tcp133 [13929.793281] LNet: Removed LNI 192.168.203.116@tcp134 [13929.801672] LNet: Removed LNI 192.168.203.116@tcp135 [13930.361630] Key type .llcrypt unregistered [13930.363673] Key type ._llcrypt unregistered [13951.206719] Key type ._llcrypt registered [13951.215519] Key type .llcrypt registered [13951.408604] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [13970.877771] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [13971.882902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13971.954259] alg: No test for adler32 (adler32-zlib) [13973.057617] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [13973.181352] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [13974.807531] Key type lgssc registered [13976.185538] Lustre: Echo OBD driver; http://www.lustre.org/ [14007.307884] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 06:52:05 (1788864725) [14022.260421] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [14024.418271] Key type lgssc unregistered [14024.674794] LNet: 471850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14024.682730] LNetError: 471850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14024.705359] LNet: Removed LNI 192.168.203.116@tcp [14025.366900] Key type .llcrypt unregistered [14025.368830] Key type ._llcrypt unregistered [14038.692711] Key type ._llcrypt registered [14038.695048] Key type .llcrypt registered [14038.831930] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [14039.799214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14039.935799] alg: No test for adler32 (adler32-zlib) [14040.987903] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [14041.269755] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [14042.968094] Key type lgssc registered [14044.118896] Lustre: Echo OBD driver; http://www.lustre.org/ [14053.662820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14053.693140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14054.969539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14055.000302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14055.102835] Lustre: lustre-MDT0000: new disk, initializing [14055.179559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14055.209909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14059.701546] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [14070.086234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14070.199239] Lustre: 473039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14070.243811] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14070.247676] Lustre: Skipped 1 previous similar message [14070.309562] Lustre: lustre-MDT0001: new disk, initializing [14070.390200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14070.413558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14070.431549] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14074.625592] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [14082.993416] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14089.010944] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14095.065201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14095.236967] Lustre: lustre-OST0000: new disk, initializing [14095.240277] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14095.244443] Lustre: 475009:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14095.312979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14101.752295] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [14103.097657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14103.111324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14103.245558] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14112.797732] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14117.235556] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing lnet_if_list [14122.621853] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14122.655397] LNet: Added LNI 192.168.203.116@tcp10 [8/256/0/180] [14128.250801] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14128.275946] LNet: Added LNI 192.168.203.116@tcp11 [8/256/0/180] [14132.131898] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14132.178673] LNet: Added LNI 192.168.203.116@tcp12 [8/256/0/180] [14136.125744] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14136.156405] LNet: Added LNI 192.168.203.116@tcp13 [8/256/0/180] [14140.576112] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14140.600944] LNet: Added LNI 192.168.203.116@tcp14 [8/256/0/180] [14145.448699] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14145.472919] LNet: Added LNI 192.168.203.116@tcp15 [8/256/0/180] [14149.816513] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14149.859184] LNet: Added LNI 192.168.203.116@tcp16 [8/256/0/180] [14154.768495] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14154.800245] LNet: Added LNI 192.168.203.116@tcp17 [8/256/0/180] [14159.693959] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14159.726773] LNet: Added LNI 192.168.203.116@tcp18 [8/256/0/180] [14166.494828] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14166.566196] LNet: Added LNI 192.168.203.116@tcp19 [8/256/0/180] [14173.431744] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14173.475831] LNet: Added LNI 192.168.203.116@tcp20 [8/256/0/180] [14179.873554] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14179.901072] LNet: Added LNI 192.168.203.116@tcp21 [8/256/0/180] [14184.414727] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14184.449867] LNet: Added LNI 192.168.203.116@tcp22 [8/256/0/180] [14190.271921] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14190.293800] LNet: Added LNI 192.168.203.116@tcp23 [8/256/0/180] [14195.714379] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14195.753510] LNet: Added LNI 192.168.203.116@tcp24 [8/256/0/180] [14200.795216] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14200.831441] LNet: Added LNI 192.168.203.116@tcp25 [8/256/0/180] [14205.974639] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14206.008547] LNet: Added LNI 192.168.203.116@tcp26 [8/256/0/180] [14212.053834] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14212.106837] LNet: Added LNI 192.168.203.116@tcp27 [8/256/0/180] [14218.033156] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14218.063056] LNet: Added LNI 192.168.203.116@tcp28 [8/256/0/180] [14223.792280] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14223.822846] LNet: Added LNI 192.168.203.116@tcp29 [8/256/0/180] [14230.089937] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14230.121982] LNet: Added LNI 192.168.203.116@tcp30 [8/256/0/180] [14235.995376] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14236.038309] LNet: Added LNI 192.168.203.116@tcp31 [8/256/0/180] [14244.560364] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14244.604696] LNet: Added LNI 192.168.203.116@tcp32 [8/256/0/180] [14251.249746] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14251.288567] LNet: Added LNI 192.168.203.116@tcp33 [8/256/0/180] [14257.875478] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14257.908830] LNet: Added LNI 192.168.203.116@tcp34 [8/256/0/180] [14264.419673] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14264.448827] LNet: Added LNI 192.168.203.116@tcp35 [8/256/0/180] [14271.690266] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14271.730979] LNet: Added LNI 192.168.203.116@tcp36 [8/256/0/180] [14278.945930] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14278.993422] LNet: Added LNI 192.168.203.116@tcp37 [8/256/0/180] [14285.518691] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14285.550216] LNet: Added LNI 192.168.203.116@tcp38 [8/256/0/180] [14292.917645] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14292.947946] LNet: Added LNI 192.168.203.116@tcp39 [8/256/0/180] [14299.987498] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14300.022274] LNet: Added LNI 192.168.203.116@tcp40 [8/256/0/180] [14307.229656] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14307.254704] LNet: Added LNI 192.168.203.116@tcp41 [8/256/0/180] [14314.864535] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14314.885672] LNet: Added LNI 192.168.203.116@tcp42 [8/256/0/180] [14324.412917] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14324.447699] LNet: Added LNI 192.168.203.116@tcp43 [8/256/0/180] [14332.391838] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14332.425706] LNet: Added LNI 192.168.203.116@tcp44 [8/256/0/180] [14340.140402] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14340.196352] LNet: Added LNI 192.168.203.116@tcp45 [8/256/0/180] [14348.909928] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14348.939853] LNet: Added LNI 192.168.203.116@tcp46 [8/256/0/180] [14358.705961] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14358.732851] LNet: Added LNI 192.168.203.116@tcp47 [8/256/0/180] [14369.214880] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14369.253130] LNet: Added LNI 192.168.203.116@tcp48 [8/256/0/180] [14377.788933] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14377.832888] LNet: Added LNI 192.168.203.116@tcp49 [8/256/0/180] [14386.372943] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14386.404891] LNet: Added LNI 192.168.203.116@tcp50 [8/256/0/180] [14408.164434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14408.170376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14408.197552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14408.672808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14408.699846] Lustre: Skipped 1 previous similar message [14408.712241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14408.723748] Lustre: Skipped 1 previous similar message [14411.829600] Lustre: server umount lustre-MDT0000 complete [14415.251816] LustreError: 473031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788865134 with bad export cookie 3730521958038661336 [14415.262608] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14415.268094] LustreError: 473031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14415.699735] Lustre: server umount lustre-MDT0001 complete [14434.272684] Lustre: 472312:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788865137/real 1788865137] req@000000002e828e61 x1875760654756480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788865153 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14434.299983] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14434.412718] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [14436.389200] Lustre: 472312:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788865139/real 1788865139] req@00000000ef21e096 x1875760654756608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788865155 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14439.391149] Lustre: 472310:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788865142/real 1788865142] req@00000000d7dbdbd9 x1875760654756736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788865158 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14441.439349] Lustre: 472311:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788865144/real 1788865144] req@00000000196a1ac7 x1875760654756992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788865160 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14443.574654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14447.583239] Lustre: 472312:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788865150/real 1788865150] req@000000002389249e x1875760654757120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788865166 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14461.298548] LustreError: 475003:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14461.327417] LustreError: 475003:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [14461.396874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14463.456580] LustreError: 475004:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14463.488914] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14464.484556] LustreError: 475003:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14466.192061] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [14469.609856] LustreError: 475004:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14474.721140] LustreError: 475003:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14474.976919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14475.406355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14479.488427] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14479.591659] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [14487.981478] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14493.196767] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14512.611315] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14512.616793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14512.629734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14515.693347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14515.706546] Lustre: Skipped 1 previous similar message [14518.151472] Lustre: server umount lustre-OST0000 complete [14525.924099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14525.927135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14525.937099] Lustre: Skipped 2 previous similar messages [14525.948135] Lustre: Skipped 1 previous similar message [14527.842319] Lustre: server umount lustre-MDT0000 complete [14530.816200] LustreError: 473031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788865249 with bad export cookie 3730521958038663968 [14530.817836] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14530.828808] LustreError: 473031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14531.145856] Lustre: server umount lustre-MDT0001 complete [14541.279770] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [14544.453497] Key type lgssc unregistered [14544.798171] LNet: 502061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14544.806216] LNetError: 502061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14544.816308] LNet: Removed LNI 192.168.203.116@tcp [14544.822765] LNet: Removed LNI 192.168.203.116@tcp10 [14544.825941] LNet: Removed LNI 192.168.203.116@tcp11 [14544.828589] LNet: Removed LNI 192.168.203.116@tcp12 [14544.831599] LNet: Removed LNI 192.168.203.116@tcp13 [14544.834537] LNet: Removed LNI 192.168.203.116@tcp14 [14544.838986] LNet: Removed LNI 192.168.203.116@tcp15 [14544.842769] LNet: Removed LNI 192.168.203.116@tcp16 [14544.846473] LNet: Removed LNI 192.168.203.116@tcp17 [14544.848608] LNet: Removed LNI 192.168.203.116@tcp18 [14544.852415] LNet: Removed LNI 192.168.203.116@tcp19 [14544.857197] LNet: Removed LNI 192.168.203.116@tcp20 [14544.861269] LNet: Removed LNI 192.168.203.116@tcp21 [14544.868340] LNet: Removed LNI 192.168.203.116@tcp22 [14544.874270] LNet: Removed LNI 192.168.203.116@tcp23 [14544.878786] LNet: Removed LNI 192.168.203.116@tcp24 [14544.882438] LNet: Removed LNI 192.168.203.116@tcp25 [14544.899178] LNet: Removed LNI 192.168.203.116@tcp26 [14544.902532] LNet: Removed LNI 192.168.203.116@tcp27 [14544.909619] LNet: Removed LNI 192.168.203.116@tcp28 [14544.915565] LNet: Removed LNI 192.168.203.116@tcp29 [14544.920218] LNet: Removed LNI 192.168.203.116@tcp30 [14544.926271] LNet: Removed LNI 192.168.203.116@tcp31 [14544.934246] LNet: Removed LNI 192.168.203.116@tcp32 [14544.940961] LNet: Removed LNI 192.168.203.116@tcp33 [14544.948847] LNet: Removed LNI 192.168.203.116@tcp34 [14544.951630] LNet: Removed LNI 192.168.203.116@tcp35 [14544.954129] LNet: Removed LNI 192.168.203.116@tcp36 [14544.961735] LNet: Removed LNI 192.168.203.116@tcp37 [14544.965587] LNet: Removed LNI 192.168.203.116@tcp38 [14544.967528] LNet: Removed LNI 192.168.203.116@tcp39 [14544.977157] LNet: Removed LNI 192.168.203.116@tcp40 [14544.980954] LNet: Removed LNI 192.168.203.116@tcp41 [14544.983986] LNet: Removed LNI 192.168.203.116@tcp42 [14544.988240] LNet: Removed LNI 192.168.203.116@tcp43 [14544.991227] LNet: Removed LNI 192.168.203.116@tcp44 [14544.994165] LNet: Removed LNI 192.168.203.116@tcp45 [14544.997139] LNet: Removed LNI 192.168.203.116@tcp46 [14545.000438] LNet: Removed LNI 192.168.203.116@tcp47 [14545.009122] LNet: Removed LNI 192.168.203.116@tcp48 [14545.011455] LNet: Removed LNI 192.168.203.116@tcp49 [14545.020735] LNet: Removed LNI 192.168.203.116@tcp50 [14545.874903] Key type .llcrypt unregistered [14545.882653] Key type ._llcrypt unregistered [14566.128309] Key type ._llcrypt registered [14566.132636] Key type .llcrypt registered [14566.278783] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [14583.902939] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [14585.358365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14585.567404] alg: No test for adler32 (adler32-zlib) [14586.722947] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [14587.041410] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [14588.751207] Key type lgssc registered [14589.807677] Lustre: Echo OBD driver; http://www.lustre.org/ [14619.532311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14619.569859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14620.840428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14620.891459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14620.982672] Lustre: lustre-MDT0000: new disk, initializing [14621.036675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14621.052728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14624.684557] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [14637.273939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14637.377258] Lustre: 505269:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14637.435860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14637.439835] Lustre: Skipped 1 previous similar message [14637.533374] Lustre: lustre-MDT0001: new disk, initializing [14637.670386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14637.702909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14637.716658] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14641.859504] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [14650.997670] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14656.096168] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14663.359283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14663.578447] Lustre: *** cfs_fail_loc=90f, val=0*** [14663.666955] Lustre: lustre-OST0000: new disk, initializing [14663.675893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14663.681371] Lustre: 507285:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14663.766724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14669.164527] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [14671.921149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14671.928634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14672.075579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14678.387572] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14687.714457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14687.715088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14687.719278] LustreError: Skipped 1 previous similar message [14687.722824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14687.729294] Lustre: Skipped 1 previous similar message [14690.857560] Lustre: server umount lustre-OST0000 complete [14697.961323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14697.964566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14697.981217] Lustre: Skipped 1 previous similar message [14697.987588] Lustre: Skipped 1 previous similar message [14700.291205] Lustre: server umount lustre-MDT0000 complete [14703.542806] LustreError: 505261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788865422 with bad export cookie 12666798102022336556 [14703.549042] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14703.554653] LustreError: 505261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14703.790341] Lustre: server umount lustre-MDT0001 complete [14712.315547] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [14715.011502] Key type lgssc unregistered [14715.237715] LNet: 509190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14715.259318] LNetError: 509190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14715.284700] LNet: Removed LNI 192.168.203.116@tcp [14716.129378] Key type .llcrypt unregistered [14716.132342] Key type ._llcrypt unregistered [14735.118150] Key type ._llcrypt registered [14735.120232] Key type .llcrypt registered [14735.240353] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [14735.862597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14735.945852] alg: No test for adler32 (adler32-zlib) [14737.052531] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [14737.317657] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [14739.023184] Key type lgssc registered [14739.995739] Lustre: Echo OBD driver; http://www.lustre.org/ [14746.146463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14746.165163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14747.542370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14751.187210] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [14757.862790] LustreError: 510456:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14757.888434] LustreError: 510456:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [14758.692963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14758.967926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14762.980707] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [14769.855994] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14774.675321] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14780.507270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14780.815731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14786.067485] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [14793.337991] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14802.628434] loop0: detected capacity change from 0 to 204800000 [14802.792340] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14802.810856] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14802.829921] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14807.795771] loop0: detected capacity change from 0 to 204800000 [14807.916703] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14807.925786] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14807.938917] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14812.457722] loop0: detected capacity change from 0 to 204800000 [14816.933173] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14817.063503] Lustre: fs15246-OST0000: new disk, initializing [14817.072498] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14817.081579] Lustre: 514052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14817.150846] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14822.537371] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [14831.227222] loop1: detected capacity change from 0 to 204800000 [14836.136104] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14836.211128] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14836.261075] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14836.360078] Lustre: fs15246-MDT0000: new disk, initializing [14836.587606] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14836.613421] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14836.718079] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14837.728367] Lustre: 515175:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000011186e8a x1875761384434176/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788865572 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14841.175221] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [14846.895308] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14846.908688] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14846.940359] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14865.384871] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14865.423515] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14868.740848] Lustre: server umount fs15246-MDT0000 complete [14870.530785] LustreError: 510452:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14875.644406] LustreError: 512953:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14879.280825] Lustre: server umount fs15246-OST0000 complete [14890.464988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14890.479770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14890.496555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14896.111255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14896.114353] Lustre: Skipped 2 previous similar messages [14896.404762] Lustre: server umount lustre-OST0000 complete [14901.217138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14901.227479] Lustre: Skipped 1 previous similar message [14901.238750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14901.247548] Lustre: Skipped 1 previous similar message [14906.339612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14906.345656] Lustre: Skipped 1 previous similar message [14906.585472] Lustre: server umount lustre-MDT0000 complete [14910.531511] LustreError: 510436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788865629 with bad export cookie 13323488066411764741 [14910.540699] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14910.545852] LustreError: 510436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14911.178253] Lustre: server umount lustre-MDT0001 complete [14922.910100] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [14925.589409] Key type lgssc unregistered [14925.895282] LNet: 518270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14925.904899] LNetError: 518270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14926.962796] LNet: Removed LNI 192.168.203.116@tcp [14928.274326] Key type .llcrypt unregistered [14928.276368] Key type ._llcrypt unregistered [14945.020724] Key type ._llcrypt registered [14945.024935] Key type .llcrypt registered [14945.155217] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [14946.050624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14946.140077] alg: No test for adler32 (adler32-zlib) [14947.136131] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [14947.362404] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [14949.007541] Key type lgssc registered [14949.808257] Lustre: Echo OBD driver; http://www.lustre.org/ [14984.652621] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [14993.554848] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [15035.460667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15035.507792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15036.827336] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15036.873123] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15036.989108] Lustre: lustre-MDT0000: new disk, initializing [15037.109053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15037.125119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15041.921470] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [15052.734584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15052.861560] Lustre: 522733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15052.939745] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15052.944612] Lustre: Skipped 1 previous similar message [15053.032585] Lustre: lustre-MDT0001: new disk, initializing [15053.188454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15053.232956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15053.242475] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15057.416420] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [15066.410522] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15072.125985] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15080.054307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15080.386611] Lustre: lustre-OST0000: new disk, initializing [15080.392775] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15080.409564] Lustre: 524703:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15080.568874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15087.066237] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [15087.722404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15087.732908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15087.845449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15097.048537] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15101.648219] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15101.891242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15106.759576] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15107.055361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15113.699649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15113.700322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15113.716306] LustreError: Skipped 1 previous similar message [15113.722129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15113.728183] Lustre: Skipped 1 previous similar message [15113.739308] Lustre: Skipped 1 previous similar message [15115.248152] Lustre: server umount lustre-OST0000 complete [15118.817839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15118.838087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15118.846633] Lustre: Skipped 1 previous similar message [15123.941965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15123.950417] Lustre: Skipped 1 previous similar message [15124.890700] Lustre: server umount lustre-MDT0000 complete [15128.135616] LustreError: 523540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788865846 with bad export cookie 14781438983484668955 [15128.141137] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15128.153243] LustreError: 523540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15128.555956] Lustre: server umount lustre-MDT0001 complete [15154.064838] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [15164.533222] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [15193.283820] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14960 sec ========= 07:11:50 (1788865910) [15195.259245] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:11:52 (1788865912) === [15198.648670] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:11:56 (1788865916) === [15236.044620] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [15238.618538] Key type lgssc unregistered [15238.951941] LNet: 531348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15238.955748] LNetError: 531348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15238.970580] LNet: Removed LNI 192.168.203.116@tcp [15239.864281] Key type .llcrypt unregistered [15239.868238] Key type ._llcrypt unregistered