[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 492134589 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.996 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2480MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003068] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008406] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 0.009018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399996) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011142] LSM: Security Framework initializing [ 0.012038] Yama: becoming mindful. [ 0.013030] SELinux: Initializing. [ 0.015039] *** VALIDATE selinux *** [ 0.023413] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028248] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030145] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031128] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033101] *** VALIDATE tmpfs *** [ 0.034440] *** VALIDATE proc *** [ 0.035230] *** VALIDATE cgroup *** [ 0.036008] *** VALIDATE cgroup2 *** [ 0.037258] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039097] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042035] Spectre V2 : User space: Vulnerable [ 0.043009] Speculative Store Bypass: Vulnerable [ 0.046276] debug: unmapping init [mem 0xffffffffb3a59000-0xffffffffb3a60fff] [ 0.049000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049701] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050023] ... version: 2 [ 0.051010] ... bit width: 48 [ 0.052011] ... generic registers: 4 [ 0.053012] ... value mask: 0000ffffffffffff [ 0.054015] ... max period: 00007fffffffffff [ 0.055011] ... fixed-purpose events: 3 [ 0.056011] ... event mask: 000000070000000f [ 0.057300] rcu: Hierarchical SRCU implementation. [ 0.059390] smp: Bringing up secondary CPUs ... [ 0.060546] x86: Booting SMP configuration: [ 0.061022] .... node #0, CPUs: #1 #2 #3 [ 0.064216] smp: Brought up 1 node, 4 CPUs [ 0.066011] smpboot: Max logical packages: 1 [ 0.067017] smpboot: Total of 4 processors activated (19199.96 BogoMIPS) [ 0.149552] node 0 deferred pages initialised in 79ms [ 0.151099] devtmpfs: initialized [ 0.152211] x86/mm: Memory block size: 128MB [ 0.154774] gcov: version magic: 0x41383552 [ 0.156118] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158243] pinctrl core: initialized pinctrl subsystem [ 0.159163] [ 0.159688] ************************************************************* [ 0.160013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161011] ** ** [ 0.162012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163013] ** ** [ 0.164013] ** This means that this kernel is built to expose internal ** [ 0.165015] ** IOMMU data structures, which may compromise security on ** [ 0.166014] ** your system. ** [ 0.167013] ** ** [ 0.168013] ** If you see this message and you are not debugging the ** [ 0.169013] ** kernel, report this immediately to your vendor! ** [ 0.170014] ** ** [ 0.171013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172012] ************************************************************* [ 0.173820] NET: Registered protocol family 16 [ 0.174447] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.175065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.176060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.177469] cpuidle: using governor menu [ 0.180899] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.181426] PCI: Using configuration type 1 for base access [ 0.182147] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189147] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192048] cryptd: max_cpu_qlen set to 1000 [ 0.195221] ACPI: Added _OSI(Module Device) [ 0.196016] ACPI: Added _OSI(Processor Device) [ 0.197016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.198012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.202360] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.205007] ACPI: Interpreter enabled [ 0.205935] ACPI: PM: (supports S0 S3 S4 S5) [ 0.206009] ACPI: Using IOAPIC for interrupt routing [ 0.207091] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.208349] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.218019] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.219045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.220035] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.221076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.223294] acpiphp: Slot [2] registered [ 0.224135] acpiphp: Slot [5] registered [ 0.225139] acpiphp: Slot [6] registered [ 0.226118] acpiphp: Slot [7] registered [ 0.227113] acpiphp: Slot [8] registered [ 0.228098] acpiphp: Slot [9] registered [ 0.229155] acpiphp: Slot [10] registered [ 0.230102] acpiphp: Slot [3] registered [ 0.231139] acpiphp: Slot [4] registered [ 0.232095] acpiphp: Slot [11] registered [ 0.233099] acpiphp: Slot [12] registered [ 0.234122] acpiphp: Slot [13] registered [ 0.235095] acpiphp: Slot [14] registered [ 0.236101] acpiphp: Slot [15] registered [ 0.237114] acpiphp: Slot [16] registered [ 0.238163] acpiphp: Slot [17] registered [ 0.239095] acpiphp: Slot [18] registered [ 0.240093] acpiphp: Slot [19] registered [ 0.241093] acpiphp: Slot [20] registered [ 0.242097] acpiphp: Slot [21] registered [ 0.243102] acpiphp: Slot [22] registered [ 0.244133] acpiphp: Slot [23] registered [ 0.245095] acpiphp: Slot [24] registered [ 0.246096] acpiphp: Slot [25] registered [ 0.247095] acpiphp: Slot [26] registered [ 0.248102] acpiphp: Slot [27] registered [ 0.249108] acpiphp: Slot [28] registered [ 0.250120] acpiphp: Slot [29] registered [ 0.251155] acpiphp: Slot [30] registered [ 0.252100] acpiphp: Slot [31] registered [ 0.253064] PCI host bridge to bus 0000:00 [ 0.254018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.255028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.256027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.257024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.258025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.259026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.260179] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.261962] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.263332] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.269561] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.272053] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.273018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.274016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.275017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.276540] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.279864] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.282059] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.285798] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.289012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.301018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.308027] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.314383] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.392020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.403019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.422018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.434235] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.442012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.449015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.469018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.479392] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.486013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.491017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.503015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.511791] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.523061] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.530016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.547016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.559046] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.568016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.574018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.587022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.595760] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.604014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.610016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.625017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.637086] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.639344] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.641306] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.643842] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.644213] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.648142] iommu: Default domain type: Passthrough [ 0.650453] SCSI subsystem initialized [ 0.652162] ACPI: bus type USB registered [ 0.654118] usbcore: registered new interface driver usbfs [ 0.656074] usbcore: registered new interface driver hub [ 0.658075] usbcore: registered new device driver usb [ 0.660193] pps_core: LinuxPPS API ver. 1 registered [ 0.662012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.665060] PTP clock support registered [ 0.667122] EDAC MC: Ver: 3.0.0 [ 0.669512] PCI: Using ACPI for IRQ routing [ 0.671528] NetLabel: Initializing [ 0.673013] NetLabel: domain hash size = 128 [ 0.675011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.676080] NetLabel: unlabeled traffic allowed by default [ 0.678121] vgaarb: loaded [ 0.680009] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.681014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.687299] clocksource: Switched to clocksource kvm-clock [ 0.788176] VFS: Disk quotas dquot_6.6.0 [ 0.789671] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.792300] *** VALIDATE ramfs *** [ 0.793537] *** VALIDATE hugetlbfs *** [ 0.795062] pnp: PnP ACPI init [ 0.797349] pnp: PnP ACPI: found 6 devices [ 0.824021] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.827231] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.829488] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.831913] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.834260] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.836607] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.839646] NET: Registered protocol family 2 [ 0.841518] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.845539] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.849144] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.854316] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.857638] TCP: Hash tables configured (established 65536 bind 65536) [ 0.860273] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.862345] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.864911] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.868836] NET: Registered protocol family 1 [ 0.873069] RPC: Registered named UNIX socket transport module. [ 0.875212] RPC: Registered udp transport module. [ 0.876869] RPC: Registered tcp transport module. [ 0.878713] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.881140] NET: Registered protocol family 44 [ 0.882489] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.884550] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.886567] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.888959] PCI: CLS 0 bytes, default 64 [ 0.890366] Unpacking initramfs... [ 2.261341] debug: unmapping init [mem 0xffff99137cc54000-0xffff99137ffbffff] [ 2.264427] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.266412] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.269057] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 2.777620] Initialise system trusted keyrings [ 2.779520] Key type blacklist registered [ 2.780981] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.794929] zbud: loaded [ 2.833798] *** VALIDATE nfs *** [ 2.835319] *** VALIDATE nfs4 *** [ 2.838238] pstore: using deflate compression [ 2.842270] Platform Keyring initialized [ 2.949894] NET: Registered protocol family 38 [ 2.953333] Key type asymmetric registered [ 2.954982] Asymmetric key parser 'x509' registered [ 2.957286] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.959797] io scheduler mq-deadline registered [ 2.961521] io scheduler kyber registered [ 2.963756] io scheduler bfq registered [ 2.965559] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.968123] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.971318] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.973387] ACPI: Power Button [PWRF] [ 2.976861] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.982239] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.993339] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.999192] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.013338] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.042481] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.068970] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.073512] Non-volatile memory driver v1.3 [ 3.074622] Linux agpgart interface v0.103 [ 3.104079] virtio_blk virtio1: [vda] 145168 512-byte logical blocks (74.3 MB/70.9 MiB) [ 3.107102] vda: detected capacity change from 0 to 74326016 [ 3.122212] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.125339] vdb: detected capacity change from 0 to 1073741824 [ 3.138357] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.140232] vdc: detected capacity change from 0 to 2621440000 [ 3.153550] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.157058] vdd: detected capacity change from 0 to 2621440000 [ 3.172485] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.175811] vde: detected capacity change from 0 to 4294967296 [ 3.190654] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.193826] vdf: detected capacity change from 0 to 4294967296 [ 3.203408] libphy: Fixed MDIO Bus: probed [ 3.210429] usbcore: registered new interface driver usbserial_generic [ 3.212866] usbserial: USB Serial support registered for generic [ 3.215547] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.220502] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.222631] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.225180] mousedev: PS/2 mouse device common for all mice [ 3.227264] rtc_cmos 00:05: RTC can wake from S4 [ 3.229437] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.231047] rtc_cmos 00:05: registered as rtc0 [ 3.234611] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.237549] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.238668] intel_pstate: CPU model not supported [ 3.243906] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.245975] hid: raw HID events driver (C) Jiri Kosina [ 3.249173] usbcore: registered new interface driver usbhid [ 3.251073] usbhid: USB HID core driver [ 3.252640] drop_monitor: Initializing network drop monitor service [ 3.255204] Initializing XFRM netlink socket [ 3.257325] NET: Registered protocol family 10 [ 3.260204] Segment Routing with IPv6 [ 3.262336] NET: Registered protocol family 17 [ 3.265132] mpls_gso: MPLS GSO support [ 3.271126] RAS: Correctable Errors collector initialized. [ 3.272802] AVX version of gcm_enc/dec engaged. [ 3.274424] AES CTR mode by8 optimization enabled [ 3.349267] sched_clock: Marking stable (3349190482, 0)->(4305632751, -956442269) [ 3.352690] registered taskstats version 1 [ 3.354679] Loading compiled-in X.509 certificates [ 3.356709] zswap: loaded using pool lzo/zbud [ 3.377500] Key type big_key registered [ 3.386442] Key type encrypted registered [ 3.387664] ima: No TPM chip found, activating TPM-bypass! [ 3.388900] ima: Allocated hash algorithm: sha1 [ 3.389949] ima: No architecture policies found [ 3.391020] evm: Initialising EVM extended attributes: [ 3.392156] evm: security.selinux [ 3.392878] evm: security.ima [ 3.393650] evm: security.capability [ 3.394587] evm: HMAC attrs: 0x1 [ 3.396199] rtc_cmos 00:05: setting system clock to 2026-07-04 03:12:57 UTC (1783134777) [ 3.400569] debug: unmapping init [mem 0xffffffffb4a03000-0xffffffffb4bfffff] [ 3.402601] debug: unmapping init [mem 0xffffffffb3782000-0xffffffffb3a58fff] [ 3.411087] Write protecting the kernel read-only data: 28672k [ 3.414377] debug: unmapping init [mem 0xffffffffb1e03000-0xffffffffb1ffffff] [ 3.416919] debug: unmapping init [mem 0xffffffffb2714000-0xffffffffb27fffff] [ 3.446337] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.455830] systemd[1]: Detected virtualization kvm. [ 3.457777] systemd[1]: Detected architecture x86-64. [ 3.459751] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.483987] systemd[1]: No hostname configured. [ 3.485513] systemd[1]: Set hostname to . [ 3.487245] random: systemd: uninitialized urandom read (16 bytes read) [ 3.489532] systemd[1]: Initializing machine ID from random generator. [ 3.526774] random: ln: uninitialized urandom read (6 bytes read) [ 3.639418] random: systemd: uninitialized urandom read (16 bytes read) [ 3.643454] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.653984] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.658271] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Slices. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.240687] device-mapper: uevent: version 1.0.3 [ 4.242940] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.036784] random: fast init done [ 5.042798] virtio_net virtio0 ens2: renamed from eth0 [ 5.218935] scsi host0: ata_piix [ 5.282481] scsi host1: ata_piix [ 5.283578] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.285251] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.907725] random: crng init done [ 9.909269] random: 7 urandom warning(s) missed due to ratelimiting [ 9.995846] dracut-initqueue[580]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.288814] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.851478] printk: systemd: 26 output lines suppressed due to ratelimiting [ 13.285409] SELinux: Disabled at runtime. [ 13.360735] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.367931] systemd[1]: Detected virtualization kvm. [ 13.369498] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.943703] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.948431] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.956492] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.963262] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.968150] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.981687] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.991175] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ 14.133248] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 14.667849] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.029736] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.092590] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.373423] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.427469] EDAC sbridge: Ver: 1.1.2 [ 17.630914] Key type dns_resolver registered [ 17.943475] NFS: Registering the id_resolver key type [ 17.945941] Key type id_resolver registered [ 17.947507] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg203-server login: [ 57.775741] libcfs: loading out-of-tree module taints kernel. [ 57.804755] Key type ._llcrypt registered [ 57.806342] Key type .llcrypt registered [ 57.871917] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 70.246407] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 71.080766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 71.090483] alg: No test for adler32 (adler32-zlib) [ 72.352881] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 72.952600] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 74.663269] Key type lgssc registered [ 75.844257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 87.640571] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 112.019790] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 119.069763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 119.102195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 120.259204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 120.275294] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 120.341898] Lustre: lustre-MDT0000: new disk, initializing [ 120.388906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 120.402517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 122.737719] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 130.483335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 130.540347] Lustre: 6510:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 130.565401] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 130.568551] Lustre: Skipped 1 previous similar message [ 130.631320] Lustre: lustre-MDT0001: new disk, initializing [ 130.669730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 130.688056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 130.693354] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 132.867167] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 135.971748] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 141.619958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 141.777438] Lustre: lustre-OST0000: new disk, initializing [ 141.779250] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 141.781909] Lustre: 8445:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 141.824413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 143.383995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 143.390879] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 143.411589] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 145.148501] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 153.926859] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 154.042894] Lustre: lustre-OST0001: new disk, initializing [ 154.048292] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 154.057937] Lustre: 9518:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 154.112229] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 157.503723] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 164.367514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 164.371701] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 164.414744] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 165.874764] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 171.019936] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 174.049610] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing check_logdir /tmp/testlogs/ [ 176.395960] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing yml_node [ 179.244388] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 180.763968] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 182.290155] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 183.317254] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jul 3 23:15:56 EDT 2026 [ 193.076059] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 193.922541] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 207.847419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 207.856216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 207.875830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 210.400666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 210.400780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 210.409401] Lustre: Skipped 1 previous similar message [ 210.414822] Lustre: Skipped 3 previous similar messages [ 212.739601] Lustre: server umount lustre-MDT0000 complete [ 215.019582] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783134989 with bad export cookie 9438744068938215235 [ 215.020794] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 215.025697] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 215.529209] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 215.529903] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 215.538264] Lustre: Skipped 2 previous similar messages [ 220.639881] LustreError: 8452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 220.639931] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 220.652540] LustreError: 8452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 220.666932] Lustre: Skipped 2 previous similar messages [ 225.763646] LustreError: 11148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 225.765132] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 225.772706] LustreError: 11148:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 225.775750] Lustre: Skipped 1 previous similar message [ 229.343192] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 229.491494] Lustre: server umount lustre-MDT0001 complete [ 232.018167] Lustre: server umount lustre-OST0000 complete [ 234.561483] Lustre: server umount lustre-OST0001 complete [ 239.205466] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 246.620539] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 270.483222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 270.636909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 270.650653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 270.696784] Lustre: lustre-MDT0000: new disk, initializing [ 270.766100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 270.779740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 270.838034] hrtimer: interrupt took 9022467 ns [ 273.193550] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 280.530432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 280.614518] Lustre: 15065:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 280.660747] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 280.664467] Lustre: Skipped 1 previous similar message [ 280.718450] Lustre: lustre-MDT0001: new disk, initializing [ 280.818770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 280.845916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 280.855846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 283.676824] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 290.790630] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 297.173436] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 306.353850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 306.785126] Lustre: lustre-OST0000: new disk, initializing [ 306.789819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 306.795379] Lustre: 17031:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 307.029985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 308.384102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 308.388976] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 308.472668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 313.363438] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 327.514910] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 335.468746] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 335.929768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 346.796359] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 347.392945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 356.322534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 356.338083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 356.388951] Lustre: Skipped 1 previous similar message [ 356.389283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 361.031632] Lustre: server umount lustre-OST0000 complete [ 366.253060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.270245] Lustre: Skipped 1 previous similar message [ 372.786503] Lustre: server umount lustre-MDT0000 complete [ 376.808421] LustreError: 17125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 376.853356] LustreError: 17125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 377.518990] LustreError: 17030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783135151 with bad export cookie 9438744068938216761 [ 377.524616] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 377.535555] LustreError: 17030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 377.840911] Lustre: server umount lustre-MDT0001 complete [ 383.351208] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 23:19:15 (1783135155) [ 385.484682] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 387.435761] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 23:19:20 (1783135160) [ 393.876216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.294065] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 394.367661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 398.911353] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 399.844083] LustreError: 19449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 408.988699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 409.061309] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 409.109129] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 414.800753] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 425.211386] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 430.583424] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 438.177846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 444.965190] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 451.709716] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 464.361359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 464.364597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 464.365522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 464.365527] Lustre: Skipped 5 previous similar messages [ 464.376124] Lustre: Skipped 1 previous similar message [ 468.626238] Lustre: server umount lustre-OST0000 complete [ 478.459442] Lustre: server umount lustre-MDT0000 complete [ 479.718887] LustreError: 21352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 479.754664] LustreError: 21352:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 481.871900] LustreError: 19429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783135255 with bad export cookie 9438744068938217650 [ 481.874978] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 481.877389] LustreError: 19429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 482.215264] Lustre: server umount lustre-MDT0001 complete [ 492.210928] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 494.929380] Key type lgssc unregistered [ 495.196662] LNet: 22979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 495.206570] LNetError: 22979:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 495.243292] LNet: Removed LNI 192.168.202.103@tcp [ 496.045160] Key type .llcrypt unregistered [ 496.047173] Key type ._llcrypt unregistered [ 517.120395] Key type ._llcrypt registered [ 517.121671] Key type .llcrypt registered [ 517.214256] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 518.430369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 518.475035] alg: No test for adler32 (adler32-zlib) [ 519.462849] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 519.701776] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 521.335269] Key type lgssc registered [ 522.166728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 529.100563] loop: module loaded [ 529.205192] loop0: detected capacity change from 0 to 204800000 [ 529.406087] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 529.415406] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 529.708128] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 535.070531] loop0: detected capacity change from 0 to 204800000 [ 535.198102] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 535.208562] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 535.224057] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 539.720880] loop0: detected capacity change from 0 to 204800000 [ 544.616167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 544.640779] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 545.845154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 545.881596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 545.961377] Lustre: lustre-MDT0000: new disk, initializing [ 546.053972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 546.073322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 550.318047] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 559.816148] loop1: detected capacity change from 0 to 204800000 [ 565.707578] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 565.931075] Lustre: lustre-OST07c6: new disk, initializing [ 565.935634] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 565.939891] Lustre: Skipped 1 previous similar message [ 565.944306] Lustre: 26164:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 566.011296] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 571.982904] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 575.689027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 575.702848] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 575.755681] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 577.491078] Lustre: 26680:0:(osd_internal.h:1444:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 577.498830] Lustre: 26680:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 577.503851] Lustre: 26680:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 577.509563] Lustre: 26680:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 577.514498] Lustre: 26680:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 577.519509] Lustre: 26680:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 593.767404] Lustre: 25154:0:(osd_internal.h:1444:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 593.774918] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 593.781689] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 593.788125] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 593.801641] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 593.808572] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 599.203633] Lustre: 26680:0:(osd_internal.h:1444:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 599.219788] Lustre: 26680:0:(osd_internal.h:1444:osd_trans_exec_op()) Skipped 1 previous similar message [ 599.225698] Lustre: 26680:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 599.233849] Lustre: 26680:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 599.242487] Lustre: 26680:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 599.249603] Lustre: 26680:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 599.260558] Lustre: 26680:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 599.267668] Lustre: 26680:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 599.272467] Lustre: 26680:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 599.278506] Lustre: 26680:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 599.284683] Lustre: 26680:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 599.291369] Lustre: 26680:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 602.620224] Lustre: 26680:0:(osd_internal.h:1444:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 602.644443] Lustre: 26680:0:(osd_internal.h:1444:osd_trans_exec_op()) Skipped 1 previous similar message [ 602.653709] Lustre: 26680:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 602.664423] Lustre: 26680:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 602.684936] Lustre: 26680:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 602.690389] Lustre: 26680:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 602.695415] Lustre: 26680:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 602.701635] Lustre: 26680:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 602.706494] Lustre: 26680:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 602.716967] Lustre: 26680:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 602.722600] Lustre: 26680:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 602.732753] Lustre: 26680:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 608.460785] Lustre: 26680:0:(osd_internal.h:1444:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 608.470512] Lustre: 26680:0:(osd_internal.h:1444:osd_trans_exec_op()) Skipped 1 previous similar message [ 608.486342] Lustre: 26680:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 608.493506] Lustre: 26680:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 608.499742] Lustre: 26680:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 608.504201] Lustre: 26680:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 608.509715] Lustre: 26680:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 608.515087] Lustre: 26680:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 608.520700] Lustre: 26680:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 608.526520] Lustre: 26680:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 608.538274] Lustre: 26680:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 608.543644] Lustre: 26680:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 619.454749] Lustre: 25154:0:(osd_internal.h:1444:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 619.468910] Lustre: 25154:0:(osd_internal.h:1444:osd_trans_exec_op()) Skipped 5 previous similar messages [ 619.476797] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 619.492047] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 619.500327] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 619.514584] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 619.533271] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 619.537984] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 619.543939] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 619.559490] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 619.570071] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 619.577806] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 668.128384] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 668.135693] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 668.164257] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 672.156667] Lustre: server umount lustre-OST07c6 complete [ 681.742838] Lustre: server umount lustre-MDT0000 complete [ 697.778225] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 700.695431] Key type lgssc unregistered [ 701.057678] LNet: 28724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 701.066683] LNetError: 28724:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 701.092604] LNet: Removed LNI 192.168.202.103@tcp [ 701.978093] Key type .llcrypt unregistered [ 701.979548] Key type ._llcrypt unregistered [ 721.072354] Key type ._llcrypt registered [ 721.074598] Key type .llcrypt registered [ 721.182737] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 722.693830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 722.884079] alg: No test for adler32 (adler32-zlib) [ 724.154132] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 724.468937] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 726.175454] Key type lgssc registered [ 727.594174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 736.894780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 736.930578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 738.510835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 743.365557] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 748.016437] LustreError: 29997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 748.033884] LustreError: 29997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 752.478225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 753.077933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 757.593257] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 765.542337] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 771.952976] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 779.714851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 780.074812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 783.152957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 787.186887] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 795.839333] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 819.174536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 819.183752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 819.194842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 819.195192] Lustre: Skipped 1 previous similar message [ 819.233269] LustreError: Skipped 1 previous similar message [ 824.292400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 824.302572] Lustre: Skipped 2 previous similar messages [ 824.419548] Lustre: server umount lustre-OST0000 complete [ 829.409053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 829.427693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 829.432355] Lustre: Skipped 1 previous similar message [ 834.535101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 834.549920] Lustre: Skipped 1 previous similar message [ 834.710679] Lustre: server umount lustre-MDT0000 complete [ 838.721072] LustreError: 29977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783135612 with bad export cookie 3889811527265266255 [ 838.723351] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 838.741280] LustreError: 29977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 839.139779] Lustre: server umount lustre-MDT0001 complete [ 851.240052] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 854.573603] Key type lgssc unregistered [ 854.974809] LNet: 33736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.994796] LNetError: 33736:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.037915] LNet: Removed LNI 192.168.202.103@tcp [ 856.882971] Key type .llcrypt unregistered [ 856.884811] Key type ._llcrypt unregistered [ 879.773939] Key type ._llcrypt registered [ 879.777359] Key type .llcrypt registered [ 879.950819] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 881.526484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 881.797620] alg: No test for adler32 (adler32-zlib) [ 883.050827] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 883.412282] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 885.205086] Key type lgssc registered [ 886.631409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 896.167829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 896.193837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 898.062990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 903.101905] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 908.263046] LustreError: 35009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 908.310974] LustreError: 35009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 913.324630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 913.377033] LustreError: 35008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 913.749173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 920.310612] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 930.728904] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 938.416531] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 946.899888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 947.492338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 954.426364] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 963.520989] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 967.943506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 979.425492] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 979.441195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 979.467096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 983.520692] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 983.524940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 983.532572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 983.570967] Lustre: Skipped 1 previous similar message [ 985.069109] Lustre: server umount lustre-OST0000 complete [ 990.694411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 990.702131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 990.711741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 993.777867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.795908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 993.808348] Lustre: Skipped 1 previous similar message [ 995.759432] Lustre: server umount lustre-MDT0000 complete [ 998.881100] LustreError: 35937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 998.896574] LustreError: 35937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1001.567468] LustreError: 34989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783135775 with bad export cookie 12110284913784682813 [ 1001.578718] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1001.587501] LustreError: 34989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1002.274413] Lustre: server umount lustre-MDT0001 complete [ 1019.378980] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1023.289158] Key type lgssc unregistered [ 1023.705743] LNet: 38567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.711996] LNetError: 38567:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.731487] LNet: Removed LNI 192.168.202.103@tcp [ 1024.997368] Key type .llcrypt unregistered [ 1025.000082] Key type ._llcrypt unregistered [ 1051.849638] Key type ._llcrypt registered [ 1051.852574] Key type .llcrypt registered [ 1051.964914] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1052.988241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1053.114265] alg: No test for adler32 (adler32-zlib) [ 1054.204543] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 1054.470663] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1056.167585] Key type lgssc registered [ 1057.305884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1065.175464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1065.215338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1066.742173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1071.331706] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1076.210870] LustreError: 39986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1076.225207] LustreError: 39986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1078.763780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1079.087025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1083.123421] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1091.024822] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1098.765910] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1105.841856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1106.082994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1111.249416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1111.985580] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1118.374069] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1129.953655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1129.961446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1129.981915] Lustre: Skipped 2 previous similar messages [ 1129.996041] Lustre: Skipped 2 previous similar messages [ 1131.478310] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.3@tcp (stopping) [ 1134.847242] Lustre: server umount lustre-MDT0000 complete [ 1135.084892] LustreError: 41040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1135.130625] LustreError: 41040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1136.571352] LustreError: 39983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1138.716800] LustreError: 39967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783135912 with bad export cookie 9800522995175531072 [ 1138.727064] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1138.747745] LustreError: 39967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1140.193135] LustreError: 41040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1140.196391] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1140.245217] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1140.255390] LustreError: 40716:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000bd88d954 x1869753434011648/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1143.201307] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1143.210077] Lustre: Skipped 1 previous similar message [ 1146.825261] LustreError: 41040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1146.863398] LustreError: 41040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1148.329622] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1148.334560] Lustre: Skipped 1 previous similar message [ 1152.992758] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1153.329617] Lustre: server umount lustre-MDT0001 complete [ 1169.137908] Lustre: server umount lustre-OST0000 complete [ 1188.307784] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1191.321577] Key type lgssc unregistered [ 1191.633867] LNet: 43849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1191.644395] LNetError: 43849:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1191.669145] LNet: Removed LNI 192.168.202.103@tcp [ 1192.698400] Key type .llcrypt unregistered [ 1192.700767] Key type ._llcrypt unregistered [ 1217.384615] Key type ._llcrypt registered [ 1217.392584] Key type .llcrypt registered [ 1217.530484] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1218.763678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1218.866875] alg: No test for adler32 (adler32-zlib) [ 1219.985894] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 1220.235992] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1221.895296] Key type lgssc registered [ 1223.452852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1230.955630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1231.008145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1232.526685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1237.289136] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1242.086746] LustreError: 45265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1242.109454] LustreError: 45265:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1246.586051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1247.004170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1253.316440] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1260.805656] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1266.747821] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1275.263275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1275.711328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1277.750312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1281.998860] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1290.508800] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1298.399453] Lustre: lustre-OST0000: Not available for connect from 192.168.202.3@tcp (stopping) [ 1298.429247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1298.445490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1301.476413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1301.478631] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1301.480056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1301.486476] Lustre: Skipped 2 previous similar messages [ 1303.487420] Lustre: lustre-OST0000: Not available for connect from 192.168.202.3@tcp (stopping) [ 1303.725789] Lustre: server umount lustre-OST0000 complete [ 1322.417959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1322.440213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1322.455038] Lustre: Skipped 1 previous similar message [ 1322.477582] Lustre: Skipped 1 previous similar message [ 1328.871190] Lustre: server umount lustre-MDT0000 complete [ 1330.656717] LustreError: 47754:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1330.677193] LustreError: 47754:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1333.222562] LustreError: 45246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783136107 with bad export cookie 5337456205588186964 [ 1333.233288] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1339.686292] Lustre: server umount lustre-MDT0001 complete [ 1352.098689] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1354.669869] Key type lgssc unregistered [ 1355.156463] LNet: 48974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1355.165202] LNetError: 48974:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1355.195280] LNet: Removed LNI 192.168.202.103@tcp [ 1356.088107] Key type .llcrypt unregistered [ 1356.089884] Key type ._llcrypt unregistered [ 1378.374609] Key type ._llcrypt registered [ 1378.377879] Key type .llcrypt registered [ 1378.524440] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1379.702955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1379.733395] alg: No test for adler32 (adler32-zlib) [ 1380.751909] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 1380.954546] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1382.615924] Key type lgssc registered [ 1383.789882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1391.812914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1391.825823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1393.517531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1397.900598] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1403.876897] LustreError: 50387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1403.891107] LustreError: 50387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1406.663718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1407.020506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1411.887726] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1420.106843] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1426.494545] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1434.652941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1435.130549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1438.196320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1441.432247] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1447.890988] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1457.079318] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1460.414264] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1472.371237] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-0323h37m24s [ 1473.714403] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1476.060837] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.3@tcp (stopping) [ 1477.112129] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.3@tcp (stopping) [ 1478.113710] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1478.131992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1478.136178] Lustre: Skipped 2 previous similar messages [ 1478.148057] Lustre: Skipped 2 previous similar messages [ 1481.679910] Lustre: server umount lustre-MDT0000 complete [ 1482.193621] LustreError: 50383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1485.287478] LustreError: 50387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1485.296815] LustreError: 50387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1485.393911] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1488.322159] LustreError: 52877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1492.639666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1492.744205] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1492.943204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1493.007369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1494.048695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1499.389317] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1504.122807] Lustre: DEBUG MARKER: Wait for df (36526) ... [ 1505.665552] Lustre: DEBUG MARKER: done [ 1513.450906] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1513.458952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1513.471639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1513.477709] Lustre: Skipped 3 previous similar messages [ 1513.952682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1516.928320] Lustre: server umount lustre-OST0000 complete [ 1524.704875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1524.708815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1524.714426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1524.714435] Lustre: Skipped 3 previous similar messages [ 1524.721815] Lustre: Skipped 2 previous similar messages [ 1528.150201] Lustre: server umount lustre-MDT0000 complete [ 1531.012092] LustreError: 53015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783136305 with bad export cookie 10818615309967286530 [ 1531.017596] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1531.022567] LustreError: 53015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1531.377295] Lustre: server umount lustre-MDT0001 complete [ 1542.108378] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1544.225754] Key type lgssc unregistered [ 1544.464742] LNet: 55313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1544.472125] LNetError: 55313:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1544.479762] LNet: Removed LNI 192.168.202.103@tcp [ 1545.385915] Key type .llcrypt unregistered [ 1545.388470] Key type ._llcrypt unregistered [ 1581.797456] Key type ._llcrypt registered [ 1581.800897] Key type .llcrypt registered [ 1581.913616] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1583.913044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1583.936168] alg: No test for adler32 (adler32-zlib) [ 1585.222495] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 1585.573328] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1587.242649] Key type lgssc registered [ 1589.087664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1598.799433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1598.824834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1600.120457] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1600.140443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1600.331472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1604.815409] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1613.676184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1613.814844] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1613.839315] Lustre: 57522:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1614.045851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1618.146098] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1625.218497] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1631.488613] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1637.764746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1637.956228] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1638.104519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1644.773680] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1646.306881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1654.212382] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1661.187469] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-0323h40m33s [ 1663.010736] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1672.285793] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1675.553588] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1675.747424] LustreError: 57530:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1696.272949] Lustre: lustre-MDT0000: Client f25af512-3e96-496a-97a2-3020c45f5b18 (at 192.168.202.3@tcp) reconnecting [ 1715.791131] LustreError: 57530:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1718.144870] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1720.841835] Lustre: DEBUG MARKER: done [ 1727.970462] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1727.985726] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1728.006584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1728.494477] Lustre: server umount lustre-OST0000 complete [ 1733.600680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1733.621774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1733.631381] Lustre: Skipped 1 previous similar message [ 1734.689483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1734.695240] Lustre: Skipped 1 previous similar message [ 1740.770513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1740.781755] Lustre: Skipped 1 previous similar message [ 1742.908962] Lustre: server umount lustre-MDT0000 complete [ 1747.232311] LustreError: 57514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783136521 with bad export cookie 10362820221342803051 [ 1747.249933] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1747.256695] LustreError: 57514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1747.638755] Lustre: server umount lustre-MDT0001 complete [ 1758.371701] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1761.399456] Key type lgssc unregistered [ 1761.731859] LNet: 61852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1761.742485] LNetError: 61852:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1761.761754] LNet: Removed LNI 192.168.202.103@tcp [ 1762.576372] Key type .llcrypt unregistered [ 1762.577981] Key type ._llcrypt unregistered [ 1812.816525] Key type ._llcrypt registered [ 1812.819943] Key type .llcrypt registered [ 1812.922058] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1814.330113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1814.347163] alg: No test for adler32 (adler32-zlib) [ 1815.540796] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 1815.970793] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1817.759199] Key type lgssc registered [ 1818.725559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1823.004188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1823.027208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1824.398474] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1824.445454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1824.763081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1825.831408] Lustre: Failing over lustre-MDT0000 [ 1826.126103] Lustre: server umount lustre-MDT0000 complete [ 1834.612830] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:43:27 (1783136607) [ 1842.525935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1842.988703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1848.579390] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1857.936779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1858.056193] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1858.091849] Lustre: 64656:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1858.227029] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1858.428596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1863.163538] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1873.551446] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1881.077173] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1889.308308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1889.624856] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1889.933109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1897.498456] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1897.620693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1904.701461] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1910.044945] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1918.439865] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1918.451321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1918.469497] Lustre: Skipped 2 previous similar messages [ 1918.482072] Lustre: Skipped 1 previous similar message [ 1923.562359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1923.584186] Lustre: Skipped 2 previous similar messages [ 1928.674915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1928.680952] Lustre: Skipped 2 previous similar messages [ 1931.231160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1931.482723] Lustre: server umount lustre-MDT0000 complete [ 1934.595049] LustreError: 65407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783136708 with bad export cookie 8954331723927472677 [ 1934.605551] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1934.616137] LustreError: 65407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1934.911562] Lustre: server umount lustre-MDT0001 complete [ 1938.328617] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1942.836680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1951.191882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1955.295819] Lustre: 63579:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783136713/real 1783136713] req@00000000aa9a1e44 x1869754233038976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783136729 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1955.333133] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1956.383372] Lustre: 63579:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783136714/real 1783136714] req@0000000041d2ccfb x1869754233039104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783136730 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1959.808156] LustreError: 66508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1959.831841] LustreError: 66508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1959.895789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1960.027748] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1960.935029] Lustre: 63576:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783136718/real 1783136718] req@0000000046cca16d x1869754233039488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783136734 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1960.954306] LustreError: 66509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1964.070825] LustreError: 66508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1964.075121] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1965.089774] Lustre: 63579:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783136723/real 1783136723] req@00000000ab141292 x1869754233039616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783136739 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1965.154109] Lustre: 63579:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1965.599960] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1970.148367] LustreError: 68429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1970.182403] LustreError: 68429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1970.256127] Lustre: 63579:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783136728/real 1783136728] req@00000000db46840c x1869754233039744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783136744 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1974.516940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1974.872929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1979.899780] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1981.156732] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 1988.923182] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1993.681791] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2005.474281] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.477065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2005.491520] Lustre: Skipped 1 previous similar message [ 2005.491786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2005.520359] Lustre: Skipped 2 previous similar messages [ 2007.001752] Lustre: server umount lustre-MDT0000 complete [ 2010.603638] LustreError: 69590:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.633776] LustreError: 69590:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2011.404495] LustreError: 64649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783136785 with bad export cookie 8954331723927474952 [ 2011.420020] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2011.831275] Lustre: server umount lustre-MDT0001 complete [ 2018.866994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2028.825055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2031.071562] Lustre: 63579:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783136789/real 1783136789] req@00000000b1497a72 x1869754233067520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783136805 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2031.092693] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2031.116866] Lustre: Skipped 1 previous similar message [ 2036.257084] LustreError: 63575:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a761eac1 x1869754233068544/t0(0) o250->MGC192.168.202.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2036.646371] LustreError: 66508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2036.794126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2036.885657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2041.316399] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2049.421340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2054.005840] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2055.162446] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2060.323358] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2061.139781] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2066.554522] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2075.619579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2075.626648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2075.649336] Lustre: Skipped 2 previous similar messages [ 2075.649814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2075.678584] Lustre: Skipped 2 previous similar messages [ 2080.109981] Lustre: server umount lustre-MDT0000 complete [ 2080.743638] LustreError: 72601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2080.764707] LustreError: 72601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2084.214344] LustreError: 64648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783136858 with bad export cookie 8954331723927476331 [ 2084.218845] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2084.243733] LustreError: 64648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2084.696701] Lustre: server umount lustre-MDT0001 complete [ 2091.099953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2094.432563] Lustre: DEBUG MARKER: files compared the same [ 2106.702770] Lustre: server umount lustre-OST0000 complete [ 2122.463738] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2125.118374] Key type lgssc unregistered [ 2125.361608] LNet: 74390:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2125.374802] LNetError: 74390:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2125.391316] LNet: Removed LNI 192.168.202.103@tcp [ 2126.074581] Key type .llcrypt unregistered [ 2126.076763] Key type ._llcrypt unregistered [ 2148.728198] Key type ._llcrypt registered [ 2148.730661] Key type .llcrypt registered [ 2148.863418] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2149.991639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2150.029298] alg: No test for adler32 (adler32-zlib) [ 2151.147930] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 2151.440281] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2153.098305] Key type lgssc registered [ 2154.295659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2162.295977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2162.312251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2163.689476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2167.345246] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug +malloc all [ 2172.917172] LustreError: 75806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.932723] LustreError: 75806:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2175.110403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2175.442310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2179.724382] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug +malloc all [ 2186.989093] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2192.609706] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2200.431597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2200.641908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2206.657436] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug +malloc all [ 2208.822159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2214.654074] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2229.217136] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2229.231290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2229.258540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2230.241854] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2230.248191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2230.262766] Lustre: Skipped 1 previous similar message [ 2232.621683] Lustre: server umount lustre-OST0000 complete [ 2236.897257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2236.910894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2236.934803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2240.480781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2240.481529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2245.602692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2245.618370] Lustre: Skipped 2 previous similar messages [ 2251.233475] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2251.376413] Lustre: server umount lustre-MDT0000 complete [ 2255.367262] LustreError: 75789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783137029 with bad export cookie 11915190020507906440 [ 2255.383548] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2255.901629] Lustre: server umount lustre-MDT0001 complete [ 2267.860261] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2270.767378] Key type lgssc unregistered [ 2271.061731] LNet: 79339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2271.066645] LNetError: 79339:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2271.082172] LNet: Removed LNI 192.168.202.103@tcp [ 2271.856400] Key type .llcrypt unregistered [ 2271.858132] Key type ._llcrypt unregistered [ 2299.065685] Key type ._llcrypt registered [ 2299.067184] Key type .llcrypt registered [ 2299.227376] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2301.415207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2301.447071] alg: No test for adler32 (adler32-zlib) [ 2302.630453] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 2302.861699] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2304.511193] Key type lgssc registered [ 2305.746689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2314.842565] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2314.868032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2330.528224] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2335.650047] LustreError: 80731:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 4: rc = -110 [ 2362.335517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2369.344373] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2376.677989] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2384.068325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2384.253478] Lustre: *** cfs_fail_loc=706, val=0*** [ 2384.570421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.687171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2389.571635] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2389.985616] LustreError: 80756:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2389.999147] LustreError: 80756:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2395.111426] LustreError: 80757:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2397.162676] LustreError: 80756:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2397.938299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2398.275568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2403.292482] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2411.048961] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2417.172728] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2420.716560] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2420.730620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2420.760859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2423.780529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2423.784668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2423.790250] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2423.815347] Lustre: Skipped 1 previous similar message [ 2425.650560] Lustre: server umount lustre-OST0000 complete [ 2434.016454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2434.021257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2434.030979] Lustre: Skipped 1 previous similar message [ 2439.136697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2439.141470] Lustre: Skipped 2 previous similar messages [ 2443.234732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2443.253343] Lustre: Skipped 1 previous similar message [ 2444.255301] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2444.509453] Lustre: server umount lustre-MDT0000 complete [ 2448.198590] LustreError: 80737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783137222 with bad export cookie 11654021562571607982 [ 2448.204271] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2448.209791] LustreError: 80737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2448.557330] Lustre: server umount lustre-MDT0001 complete [ 2463.167979] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2466.588953] Key type lgssc unregistered [ 2467.097713] LNet: 84348:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2467.115215] LNetError: 84348:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2467.133179] LNet: Removed LNI 192.168.202.103@tcp [ 2468.586512] Key type .llcrypt unregistered [ 2468.587935] Key type ._llcrypt unregistered [ 2496.288617] Key type ._llcrypt registered [ 2496.293883] Key type .llcrypt registered [ 2496.504747] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2498.540926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2498.864931] alg: No test for adler32 (adler32-zlib) [ 2500.478627] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 2500.830169] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2502.463208] Key type lgssc registered [ 2503.666315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2511.540666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2511.558157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2516.919796] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2524.379402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2524.952763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2529.325393] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2530.279372] LustreError: 86455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2530.300612] LustreError: 86455:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2534.370276] LustreError: 86459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2538.246138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2538.959549] LustreError: 86453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2538.991434] LustreError: 86453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2539.012361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2544.097382] LustreError: 86458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2544.130190] LustreError: 86458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2545.649781] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2549.230360] LustreError: 86454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.267231] LustreError: 86454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2556.009670] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2556.031933] Lustre: MGS: Client 14188a0b-43c8-46fc-8566-74601289f704 (at 0@lo) reconnecting [ 2556.070697] LustreError: 86469:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2556.367152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2556.445580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2561.775354] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2576.869536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2576.871725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2576.899982] Lustre: Skipped 1 previous similar message [ 2581.789455] Lustre: server umount lustre-OST0000 complete [ 2587.119116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2587.125662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2587.127211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2587.138774] Lustre: Skipped 1 previous similar message [ 2587.167673] Lustre: Skipped 2 previous similar messages [ 2592.148354] Lustre: server umount lustre-MDT0000 complete [ 2592.226286] LustreError: 86458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2592.236641] LustreError: 86458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2597.910646] Lustre: server umount lustre-MDT0001 complete [ 2602.821892] Lustre: server umount MGS complete [ 2616.965924] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 23:56:29 (1783137389) [ 2646.899526] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 2656.448586] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2696.745699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2702.094731] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2713.373011] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2713.791620] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2713.884131] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2714.051707] Lustre: lustre-MDT0001: new disk, initializing [ 2714.190558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2714.237732] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2714.251393] Lustre: 93655:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2714.273186] LustreError: 93655:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2718.234849] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2729.244804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2729.669125] Lustre: lustre-OST0000: new disk, initializing [ 2729.672643] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2729.679259] Lustre: 94421:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2729.761917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2737.276983] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2739.809390] Lustre: 94827:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000008e2f6707 x1869754950262784/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1783137529 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2749.408642] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2757.048571] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2757.081901] Lustre: MGS: Client 161f5245-ca97-4e97-90c1-857f69d5a7cc (at 0@lo) reconnecting [ 2757.157505] Lustre: 92919:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2757.182804] LustreError: 93641:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2757.230460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2757.391688] Lustre: lustre-MDT0000: new disk, initializing [ 2757.743057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2757.793759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2757.822951] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2758.880896] Lustre: 95490:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000fb926f92 x1869754950269696/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1783137548 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2763.589805] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2764.148290] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2767.088582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2767.095660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2768.257883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2788.859056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2788.873656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2788.888104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2792.434512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2792.442409] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2794.960872] Lustre: server umount lustre-OST0000 complete [ 2802.662051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2802.666468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2802.677215] Lustre: Skipped 1 previous similar message [ 2802.678265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2802.678271] Lustre: Skipped 1 previous similar message [ 2804.743070] Lustre: server umount lustre-MDT0000 complete [ 2807.785569] LustreError: 93631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2807.815297] LustreError: 93631:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2815.034348] Lustre: server umount lustre-MDT0001 complete [ 2819.022714] Lustre: server umount MGS complete [ 2833.506699] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 00:00:05 (1783137605) [ 2856.471793] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2859.931396] Key type lgssc unregistered [ 2860.260646] LNet: 97992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2860.265471] LNetError: 97992:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2860.296569] LNet: Removed LNI 192.168.202.103@tcp [ 2861.459973] Key type .llcrypt unregistered [ 2861.461764] Key type ._llcrypt unregistered [ 2881.699361] Key type ._llcrypt registered [ 2881.710551] Key type .llcrypt registered [ 2881.873636] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2883.991049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2884.033263] alg: No test for adler32 (adler32-zlib) [ 2885.386913] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 2885.810221] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2887.615133] Key type lgssc registered [ 2889.822323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2898.832746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2898.851034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2900.441449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2905.244555] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2910.705478] LustreError: 99227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2910.728854] LustreError: 99227:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2914.824590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2915.119375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2920.496171] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2929.642221] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2935.556948] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2943.441374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2943.799609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2945.851719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2951.186567] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 2958.913731] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2976.751224] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2976.756590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2976.760850] LustreError: Skipped 1 previous similar message [ 2976.768691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2976.782344] Lustre: Skipped 1 previous similar message [ 2976.802743] Lustre: Skipped 1 previous similar message [ 2978.054979] Lustre: server umount lustre-OST0000 complete [ 2981.857725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2981.859748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2981.874086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2981.909946] Lustre: Skipped 1 previous similar message [ 2986.979578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2986.983630] Lustre: Skipped 1 previous similar message [ 2987.978056] Lustre: server umount lustre-MDT0000 complete [ 2991.526227] LustreError: 99210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783137765 with bad export cookie 16167139310418276586 [ 2991.530037] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2991.543453] LustreError: 99210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2991.897407] Lustre: server umount lustre-MDT0001 complete [ 3004.631714] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 3007.943780] Key type lgssc unregistered [ 3008.371756] LNet: 102783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3008.375223] LNetError: 102783:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3008.395756] LNet: Removed LNI 192.168.202.103@tcp [ 3009.328252] Key type .llcrypt unregistered [ 3009.329985] Key type ._llcrypt unregistered [ 3034.371795] Key type ._llcrypt registered [ 3034.379199] Key type .llcrypt registered [ 3034.567247] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3035.975592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3036.113895] alg: No test for adler32 (adler32-zlib) [ 3037.745438] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 3038.080499] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 3039.799417] Key type lgssc registered [ 3041.469614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3049.120263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3049.132522] LustreError: 103866:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3049.151889] LustreError: 103869:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3049.158046] LustreError: 103866:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4986 [ 3049.164053] LustreError: 103866:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3049.174559] LustreError: 103866:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3049.194820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3050.706432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3058.259762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3058.495602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3062.961413] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3069.027812] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing lsmod [ 3071.981130] LustreError: 105077:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3072.000581] LustreError: 105076:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3072.006252] LustreError: 105077:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4989 [ 3072.017180] LustreError: 105077:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3072.026869] LustreError: 105077:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3072.037153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3072.474570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3075.573197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:65) [ 3079.137848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3079.148585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3079.166057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3080.675617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3080.684042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3080.713094] Lustre: Skipped 1 previous similar message [ 3082.642550] Lustre: server umount lustre-MDT0000 complete [ 3083.815405] LustreError: 104714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3083.850689] LustreError: 104714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3087.097843] LustreError: 105262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783137861 with bad export cookie 15890179271852332712 [ 3087.099206] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3087.107061] LustreError: 105262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3087.967438] LustreError: 104338:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000bfe02224 x1869755514377728/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3088.203392] Lustre: server umount lustre-MDT0001 complete [ 3102.894885] Lustre: server umount lustre-OST0000 complete [ 3112.613141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3112.946586] LustreError: 106140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3113.026037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3118.068495] LustreError: 106141:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3118.134647] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3123.171121] LustreError: 106140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3126.897995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3127.181535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3131.768651] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3139.849692] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3145.171396] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3154.516618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3154.998206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3159.119922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:97) [ 3162.973326] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3171.172523] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3179.490990] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3179.497146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3179.512737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3179.519660] Lustre: Skipped 2 previous similar messages [ 3182.560922] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3182.565127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3182.604553] Lustre: Skipped 1 previous similar message [ 3183.917160] Lustre: server umount lustre-OST0000 complete [ 3187.680787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3187.698236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3187.705526] Lustre: Skipped 1 previous similar message [ 3193.945077] Lustre: server umount lustre-MDT0000 complete [ 3197.813289] LustreError: 106122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783137971 with bad export cookie 15890179271852333650 [ 3197.819155] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3197.828895] LustreError: 106122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3198.321974] Lustre: server umount lustre-MDT0001 complete [ 3209.322152] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 3212.125967] Key type lgssc unregistered [ 3212.511416] LNet: 109675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3212.523836] LNetError: 109675:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3212.564786] LNet: Removed LNI 192.168.202.103@tcp [ 3213.604531] Key type .llcrypt unregistered [ 3213.608508] Key type ._llcrypt unregistered [ 3237.475701] Key type ._llcrypt registered [ 3237.480402] Key type .llcrypt registered [ 3237.579877] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3238.892299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3239.055588] alg: No test for adler32 (adler32-zlib) [ 3240.174226] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 3240.359585] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 3242.039207] Key type lgssc registered [ 3243.515346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3250.920841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3250.933600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3252.458511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3257.566449] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3261.924054] LustreError: 111101:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3261.939036] LustreError: 111101:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3266.409512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3266.924878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3271.591287] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3280.305295] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3285.942609] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3293.290750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3293.741860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3297.841053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3300.924929] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3309.183833] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3316.087913] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3321.398709] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3328.481193] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3328.510159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3328.547950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3331.564219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3331.564490] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3331.572541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3331.583099] Lustre: Skipped 1 previous similar message [ 3336.691861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3336.698457] Lustre: Skipped 1 previous similar message [ 3341.798261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3341.808582] Lustre: Skipped 1 previous similar message [ 3342.303138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3342.626552] Lustre: server umount lustre-OST0000 complete [ 3348.960725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3348.975252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3348.998584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3352.035167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3354.472537] Lustre: server umount lustre-MDT0000 complete [ 3357.153490] LustreError: 111102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3357.167537] LustreError: 111102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3358.228392] LustreError: 113635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138132 with bad export cookie 17410693485797725602 [ 3358.229470] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3358.239161] LustreError: 113635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3358.680880] Lustre: server umount lustre-MDT0001 complete [ 3369.602926] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 3372.784945] Key type lgssc unregistered [ 3373.190413] LNet: 114743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3373.207525] LNetError: 114743:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3374.245596] LNet: Removed LNI 192.168.202.103@tcp [ 3375.280586] Key type .llcrypt unregistered [ 3375.282636] Key type ._llcrypt unregistered [ 3393.908567] Key type ._llcrypt registered [ 3393.914561] Key type .llcrypt registered [ 3394.051384] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3395.426827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3395.452783] alg: No test for adler32 (adler32-zlib) [ 3396.449896] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 3396.689428] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 3398.407203] Key type lgssc registered [ 3399.766765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3407.063923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3407.118520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3408.658835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3414.349327] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3418.086417] LustreError: 115963:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3418.116495] LustreError: 115963:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3423.201758] LustreError: 115964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3425.519797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3426.169158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3431.791482] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3440.202809] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3446.682415] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3454.051172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3454.613625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3461.090571] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3468.337827] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3479.148521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3489.783036] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3489.787833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3489.800365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3489.811168] Lustre: Skipped 1 previous similar message [ 3494.882988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3494.896144] Lustre: Skipped 1 previous similar message [ 3500.001384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3500.012520] Lustre: Skipped 1 previous similar message [ 3501.023240] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3501.282592] Lustre: server umount lustre-OST0000 complete [ 3505.120602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3505.145528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3505.149666] Lustre: Skipped 2 previous similar messages [ 3510.243614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3510.248677] Lustre: Skipped 1 previous similar message [ 3511.295568] Lustre: server umount lustre-MDT0000 complete [ 3515.360903] LustreError: 116908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3515.386674] LustreError: 116908:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3515.730093] LustreError: 115944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138289 with bad export cookie 584281713760535289 [ 3515.733480] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3515.748092] LustreError: 115944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3516.283764] Lustre: server umount lustre-MDT0001 complete [ 3529.023689] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 3532.379107] Key type lgssc unregistered [ 3532.624987] LNet: 119526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3532.631282] LNetError: 119526:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3533.676539] LNet: Removed LNI 192.168.202.103@tcp [ 3534.934891] Key type .llcrypt unregistered [ 3534.936584] Key type ._llcrypt unregistered [ 3563.972518] Key type ._llcrypt registered [ 3563.973967] Key type .llcrypt registered [ 3564.096210] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3565.186268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3565.224370] alg: No test for adler32 (adler32-zlib) [ 3566.243784] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 3566.489482] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 3568.351210] Key type lgssc registered [ 3570.136271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3579.911032] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3579.959372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3581.644907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3588.915613] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3591.149593] LustreError: 120944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3591.179985] LustreError: 120944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3596.269277] LustreError: 120945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3598.077895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3598.659702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3603.595112] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3613.883100] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3621.950814] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3630.155388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3630.564173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3637.852910] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3645.908984] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3651.073071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3658.295132] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3662.258890] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3662.269566] Lustre: Skipped 2 previous similar messages [ 3665.025813] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3665.067220] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3666.475730] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3666.480258] Lustre: Skipped 2 previous similar messages [ 3673.721175] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3673.740059] Lustre: Skipped 1 previous similar message [ 3681.094165] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.3@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3681.103880] Lustre: Skipped 2 previous similar messages [ 3682.165254] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.3@tcp 192.168.0.[2,10]@tcp [ 3694.701331] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.3/24@tcp in log lustre-MDT0000 [ 3694.721201] Lustre: Skipped 2 previous similar messages [ 3700.838947] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.3/24@tcp [ 3700.851458] Lustre: Skipped 1 previous similar message [ 3710.439243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3710.459707] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3710.474914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3712.485532] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3712.492697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3712.496514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3712.511237] Lustre: Skipped 1 previous similar message [ 3715.632482] Lustre: server umount lustre-OST0000 complete [ 3721.702375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3721.735072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3721.775604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3725.593339] Lustre: server umount lustre-MDT0000 complete [ 3727.843543] LustreError: 121945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3727.863908] LustreError: 121945:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3731.004147] LustreError: 120925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138505 with bad export cookie 10517728936951599780 [ 3731.026838] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3731.030264] LustreError: 120925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3731.685932] Lustre: server umount lustre-MDT0001 complete [ 3745.018123] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 3748.267636] Key type lgssc unregistered [ 3748.678727] LNet: 125894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3748.701514] LNetError: 125894:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3748.733188] LNet: Removed LNI 192.168.202.103@tcp [ 3749.939246] Key type .llcrypt unregistered [ 3749.943973] Key type ._llcrypt unregistered [ 3782.505130] Key type ._llcrypt registered [ 3782.508355] Key type .llcrypt registered [ 3782.633729] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3783.849100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3784.029779] alg: No test for adler32 (adler32-zlib) [ 3785.073755] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 3785.374918] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 3787.048135] Key type lgssc registered [ 3788.287280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3795.669800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3795.687122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3797.179776] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3797.199300] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.3/24@tcp [ 3797.211298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3800.800050] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3807.721150] LustreError: 127455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3807.756394] LustreError: 127455:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3809.131794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3809.421845] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3809.464170] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.3/24@tcp [ 3809.571267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3814.875192] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3824.107813] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3831.018638] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3837.837437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3838.227568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3842.383904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:201 to 0x280000401:225) [ 3844.382453] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 3852.536860] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3865.061445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3865.071775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3865.088733] Lustre: Skipped 1 previous similar message [ 3865.110972] Lustre: Skipped 1 previous similar message [ 3869.960553] Lustre: server umount lustre-OST0000 complete [ 3875.296848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3875.328153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3875.335857] Lustre: Skipped 1 previous similar message [ 3880.073191] Lustre: server umount lustre-MDT0000 complete [ 3880.417414] LustreError: 128899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3880.433300] LustreError: 128899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3883.885486] LustreError: 127436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138657 with bad export cookie 14582784028027941460 [ 3883.886370] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3883.900602] LustreError: 127436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3884.270611] Lustre: server umount lustre-MDT0001 complete [ 3894.947231] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 3898.165035] Key type lgssc unregistered [ 3898.513103] LNet: 131043:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3898.531540] LNetError: 131043:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3899.560363] LNet: Removed LNI 192.168.202.103@tcp [ 3900.736758] Key type .llcrypt unregistered [ 3900.739396] Key type ._llcrypt unregistered [ 3928.666407] Key type ._llcrypt registered [ 3928.671935] Key type .llcrypt registered [ 3928.786878] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 3944.559214] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3945.839307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3945.866726] alg: No test for adler32 (adler32-zlib) [ 3947.019584] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 3947.317316] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 3949.023167] Key type lgssc registered [ 3950.422908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3989.566897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3989.613681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3991.108793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3991.178465] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3991.292394] Lustre: lustre-MDT0000: new disk, initializing [ 3991.386721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3991.413698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3996.719237] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4009.705411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4009.826466] Lustre: 134447:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4009.889899] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4009.894571] Lustre: Skipped 1 previous similar message [ 4010.012509] Lustre: lustre-MDT0001: new disk, initializing [ 4010.141560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4010.190505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4010.209019] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4014.979541] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4023.693422] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4029.403553] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4037.763967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4038.054133] Lustre: lustre-OST0000: new disk, initializing [ 4038.058368] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4038.087512] Lustre: 136416:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4038.187828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4044.883134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4044.893486] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4045.008996] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4045.022415] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4054.053861] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4058.847056] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4059.095804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4063.545572] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4063.751669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4070.884954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4070.890711] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4070.901683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4070.909754] Lustre: Skipped 1 previous similar message [ 4076.002869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4076.020051] Lustre: Skipped 1 previous similar message [ 4080.095151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4080.199975] Lustre: server umount lustre-OST0000 complete [ 4086.256221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4086.267878] Lustre: Skipped 1 previous similar message [ 4086.275871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4090.101947] Lustre: server umount lustre-MDT0000 complete [ 4093.198056] LustreError: 134439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138867 with bad export cookie 1331911243987591796 [ 4093.204495] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4093.207379] LustreError: 134439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4093.475439] Lustre: server umount lustre-MDT0001 complete [ 4100.911685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4101.210485] LustreError: 138631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4101.220808] LustreError: 138631:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4101.266989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4105.211918] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4106.722590] LustreError: 138632:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4111.842226] LustreError: 138631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4112.759772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4113.072722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4117.568143] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4125.316756] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4131.493465] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4138.895422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4139.226735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4145.843150] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4152.611623] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4157.044151] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4157.242571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4161.713993] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4162.000135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4170.853820] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4171.017893] Lustre: lustre-OST0001: new disk, initializing [ 4171.029905] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4171.036258] Lustre: 142001:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4171.138601] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4177.297593] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4181.101111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4181.117058] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4181.201541] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4185.084892] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4185.239807] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4190.618412] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4190.842839] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4196.055939] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e3682b0f800.ost_server_uuid 50 [ 4197.509861] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8e3682b0f800.ost_server_uuid in FULL state after 0 sec [ 4207.088733] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4207.091911] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4207.099925] Lustre: Skipped 1 previous similar message [ 4207.102856] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4207.107132] LustreError: Skipped 1 previous similar message [ 4207.110549] Lustre: Skipped 2 previous similar messages [ 4212.200587] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4212.209449] Lustre: Skipped 1 previous similar message [ 4218.335179] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4218.445207] Lustre: server umount lustre-OST0001 complete [ 4222.438393] LustreError: 140455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4222.441436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4222.472676] LustreError: 140455:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4222.473241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4222.473245] Lustre: Skipped 2 previous similar messages [ 4222.510841] Lustre: Skipped 2 previous similar messages [ 4227.567267] LustreError: 140455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4227.589190] LustreError: 140455:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4227.802681] Lustre: server umount lustre-OST0000 complete [ 4231.136880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4231.149257] LustreError: Skipped 1 previous similar message [ 4231.160075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4236.939375] Lustre: server umount lustre-MDT0000 complete [ 4237.793921] LustreError: 138627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4237.806766] LustreError: 138627:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4240.535691] LustreError: 138613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139014 with bad export cookie 1331911243987592678 [ 4240.540275] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4240.546465] LustreError: 138613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4240.892558] Lustre: server umount lustre-MDT0001 complete [ 4257.091540] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 4260.025132] Key type lgssc unregistered [ 4260.338948] LNet: 145007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4260.356353] LNetError: 145007:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4260.378294] LNet: Removed LNI 192.168.202.103@tcp [ 4261.229282] Key type .llcrypt unregistered [ 4261.231189] Key type ._llcrypt unregistered [ 4303.856540] Key type ._llcrypt registered [ 4303.860591] Key type .llcrypt registered [ 4303.992367] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 4316.158599] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 4316.989331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4317.139113] alg: No test for adler32 (adler32-zlib) [ 4318.232392] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 4318.468679] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 4320.175239] Key type lgssc registered [ 4321.185886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4349.177435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4349.210319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4350.492438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4350.530486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4350.654231] Lustre: lustre-MDT0000: new disk, initializing [ 4350.763626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4350.781748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4354.931300] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4366.492456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4366.684081] Lustre: 149140:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4366.879404] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4366.887224] Lustre: Skipped 1 previous similar message [ 4367.141721] Lustre: lustre-MDT0001: new disk, initializing [ 4367.278263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4367.329224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4367.345101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4373.512686] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4384.165915] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4389.859654] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4398.411075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4398.698908] Lustre: lustre-OST0000: new disk, initializing [ 4398.701836] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4398.707462] Lustre: 151108:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4398.792042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4405.816781] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4406.353915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4406.369905] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4406.464621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4415.073967] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4420.735160] Lustre: Failing over lustre-OST0000 [ 4420.928888] Lustre: server umount lustre-OST0000 complete [ 4438.496667] Lustre: 147368:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139196/real 1783139196] req@00000000c8d46e8e x1869756856566784/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1783139212 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4438.552921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4438.579913] Lustre: Skipped 1 previous similar message [ 4440.906412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4441.334440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4441.368574] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4443.359334] Lustre: 147367:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139201/real 1783139201] req@00000000b5231b81 x1869756856568832/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783139217 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4443.404200] Lustre: 147367:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4448.031224] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4448.494043] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4448.520130] Lustre: 147368:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139206/real 1783139206] req@00000000876a8333 x1869756856569984/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783139222 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4448.743053] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4448.745652] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4451.900291] Lustre: Failing over lustre-MDT0000 [ 4452.368670] Lustre: server umount lustre-MDT0000 complete [ 4453.861637] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4453.869442] Lustre: Skipped 1 previous similar message [ 4454.880057] Lustre: 147368:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139211/real 1783139211] req@00000000a2ffb5db x1869756856571008/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783139227 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4454.933686] Lustre: 147368:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4458.939905] LustreError: 149149:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4458.981057] LustreError: 149149:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4464.065848] LustreError: 149147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4469.181163] LustreError: 149148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4469.197538] LustreError: 149148:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4469.216263] Lustre: 147365:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139227/real 1783139227] req@00000000ee90c8c0 x1869756856580992/t0(0) o400->MGC192.168.202.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783139243 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4469.239182] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4471.720982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4474.296694] LustreError: 149148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4474.319430] LustreError: 149148:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4479.457392] LustreError: 149153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4479.472327] LustreError: 149153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4479.858430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4479.909311] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4480.443311] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4485.121444] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4485.125981] Lustre: Skipped 1 previous similar message [ 4485.161910] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4485.225727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4485.983491] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4494.816588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4494.827208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4494.837210] Lustre: Skipped 1 previous similar message [ 4494.843879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4500.454753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4500.465204] Lustre: Skipped 3 previous similar messages [ 4501.201428] Lustre: server umount lustre-OST0000 complete [ 4505.568221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4505.579387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4505.594498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4505.595130] Lustre: Skipped 2 previous similar messages [ 4505.606446] Lustre: Skipped 1 previous similar message [ 4510.688577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4510.695609] Lustre: Skipped 1 previous similar message [ 4515.812452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4515.822943] Lustre: Skipped 1 previous similar message [ 4519.392871] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4519.574399] Lustre: server umount lustre-MDT0000 complete [ 4520.928255] LustreError: 149153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4520.941928] LustreError: 149153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4523.422859] LustreError: 149133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139297 with bad export cookie 10195723218289281890 [ 4523.431122] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4523.438315] LustreError: 149133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4523.801678] Lustre: server umount lustre-MDT0001 complete [ 4536.162751] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 4538.806754] Key type lgssc unregistered [ 4539.169760] LNet: 154767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4539.196088] LNetError: 154767:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4539.227092] LNet: Removed LNI 192.168.202.103@tcp [ 4540.048706] Key type .llcrypt unregistered [ 4540.050282] Key type ._llcrypt unregistered [ 4577.382518] Key type ._llcrypt registered [ 4577.390854] Key type .llcrypt registered [ 4577.494831] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 4578.647984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4578.765138] alg: No test for adler32 (adler32-zlib) [ 4579.869674] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 4580.088899] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 4581.743286] Key type lgssc registered [ 4582.776483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4591.005158] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4591.031815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4592.622060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4596.780601] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4602.852991] LustreError: 156291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4602.874576] LustreError: 156291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4605.209930] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4607.972272] LustreError: 156292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4609.209844] LustreError: 156320:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4609.227101] LustreError: 156320:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4609.232611] LustreError: 156320:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4609.893519] Lustre: server umount lustre-MDT0000 complete [ 4625.255493] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 4635.107195] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 4670.037215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4670.238539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4670.271863] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4670.366657] Lustre: lustre-MDT0000: new disk, initializing [ 4670.449614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4670.462881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4675.015760] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4685.520631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4685.683963] Lustre: 159625:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4685.773349] LustreError: 159635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4685.813929] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4685.818877] Lustre: Skipped 1 previous similar message [ 4685.892400] Lustre: lustre-MDT0001: new disk, initializing [ 4685.980977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4686.048026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4686.057287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4689.767714] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4700.928704] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4707.145638] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4714.068643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4714.419732] Lustre: lustre-OST0000: new disk, initializing [ 4714.425149] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4714.434264] Lustre: 161591:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4714.606740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4721.355682] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 4721.751182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4721.763991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4721.944367] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4732.514139] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4737.982777] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4739.904665] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 00:31:52 (1783139512) [ 4962.784158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4962.785355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4962.794268] Lustre: Skipped 1 previous similar message [ 4962.795793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4962.812708] Lustre: Skipped 2 previous similar messages [ 4965.753685] Lustre: server umount lustre-MDT0000 complete [ 4967.908215] LustreError: 159636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4967.961643] LustreError: 159636:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4969.919484] LustreError: 159616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139744 with bad export cookie 9820088630093690481 [ 4969.920581] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4969.925207] LustreError: 159616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4973.024660] LustreError: 159632:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4973.025047] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4973.048403] Lustre: Skipped 1 previous similar message [ 4973.052075] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4979.168566] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4979.175635] Lustre: Skipped 1 previous similar message [ 4984.287221] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4984.534242] Lustre: server umount lustre-MDT0001 complete [ 4995.474554] Lustre: server umount lustre-OST0000 complete [ 5003.368763] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 5011.421387] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5044.755889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5045.070345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5045.092882] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5045.188895] Lustre: lustre-MDT0000: new disk, initializing [ 5045.301847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5045.322974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5049.501344] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5059.587763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5059.700754] Lustre: 166251:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5059.740682] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5059.744591] Lustre: Skipped 1 previous similar message [ 5059.800205] Lustre: lustre-MDT0001: new disk, initializing [ 5059.897801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5059.943605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5059.967297] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5064.554954] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5073.727290] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5080.195524] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5087.946875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5088.333244] Lustre: lustre-OST0000: new disk, initializing [ 5088.338947] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5088.345761] Lustre: 168217:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5088.519358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5095.713205] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5098.124415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5098.150567] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5098.303099] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5106.540073] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5111.319240] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5111.469591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5115.529182] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5115.797629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5118.945336] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5118.948314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5118.960199] LustreError: Skipped 1 previous similar message [ 5118.962233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5118.962242] Lustre: Skipped 1 previous similar message [ 5119.031423] Lustre: Skipped 1 previous similar message [ 5124.023967] Lustre: server umount lustre-OST0000 complete [ 5131.744700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5131.767057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5131.813550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5131.832020] Lustre: Skipped 3 previous similar messages [ 5134.901565] Lustre: server umount lustre-MDT0000 complete [ 5137.383248] LustreError: 166256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5137.399668] LustreError: 166256:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5139.230684] LustreError: 166241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139913 with bad export cookie 9820088630093849311 [ 5139.233239] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5139.260645] LustreError: 166241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5139.742575] Lustre: server umount lustre-MDT0001 complete [ 5149.203788] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 00:38:41 (1783139921) [ 5166.885940] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 5175.651532] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5215.032711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5215.274404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5215.292260] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5215.356281] Lustre: lustre-MDT0000: new disk, initializing [ 5215.420827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5215.437856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5220.749873] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5231.983038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5232.148551] Lustre: 172973:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5232.539439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5237.354295] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5246.456249] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5252.915207] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5259.729881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5259.986567] Lustre: lustre-OST0000: new disk, initializing [ 5259.988648] Lustre: Skipped 1 previous similar message [ 5259.991886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5259.998700] Lustre: Skipped 2 previous similar messages [ 5260.004299] Lustre: 174950:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5265.896312] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5266.511484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5266.523313] Lustre: Skipped 1 previous similar message [ 5266.534773] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5266.635040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5275.089934] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5282.272572] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5282.275749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5282.287153] LustreError: Skipped 1 previous similar message [ 5282.294312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5282.308225] Lustre: Skipped 2 previous similar messages [ 5282.339936] Lustre: Skipped 3 previous similar messages [ 5296.095511] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5296.442967] Lustre: server umount lustre-OST0000 complete [ 5302.754286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5302.759964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5302.767849] Lustre: Skipped 1 previous similar message [ 5302.779575] Lustre: Skipped 5 previous similar messages [ 5306.812908] Lustre: server umount lustre-MDT0000 complete [ 5307.877542] LustreError: 172981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5307.903736] LustreError: 172981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5311.114692] LustreError: 172966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140085 with bad export cookie 9820088630093850186 [ 5311.114952] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5311.136812] LustreError: 172966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5311.807425] Lustre: server umount lustre-MDT0001 complete [ 5319.165315] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 00:41:32 (1783140092) [ 5336.285771] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 5346.060555] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5393.946856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5394.215971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5394.270481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5394.388105] Lustre: lustre-MDT0000: new disk, initializing [ 5394.484178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5394.489244] Lustre: Skipped 2 previous similar messages [ 5394.510783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5400.813836] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5412.150590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5412.398668] Lustre: 179277:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5412.771539] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5418.079469] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5428.839557] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5434.048673] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5441.356401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5441.563251] Lustre: 181254:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5448.814324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5448.835883] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5448.989961] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5459.363967] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5469.671520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5469.674533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5469.690100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5473.237600] Lustre: server umount lustre-OST0000 complete [ 5479.394808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5479.410784] LustreError: Skipped 1 previous similar message [ 5491.167469] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5491.587452] Lustre: server umount lustre-MDT0000 complete [ 5494.248461] LustreError: 180081:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5494.274403] LustreError: 180081:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5495.922205] LustreError: 181265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140270 with bad export cookie 9820088630093851278 [ 5495.930399] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5495.935259] LustreError: 181265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5496.270210] Lustre: server umount lustre-MDT0001 complete [ 5506.085943] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 5508.557939] Key type lgssc unregistered [ 5508.815663] LNet: 183110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5508.821707] LNetError: 183110:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5508.833608] LNet: Removed LNI 192.168.202.103@tcp [ 5509.648680] Key type .llcrypt unregistered [ 5509.650168] Key type ._llcrypt unregistered [ 5532.395520] Key type ._llcrypt registered [ 5532.397683] Key type .llcrypt registered [ 5532.490437] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5533.912190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5533.974864] alg: No test for adler32 (adler32-zlib) [ 5535.155656] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 5535.479786] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 5537.168074] Key type lgssc registered [ 5538.464666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5546.153369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5546.167361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5547.585943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5552.708925] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5557.728445] LustreError: 184531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5557.754524] LustreError: 184531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5561.348391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5561.833450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5566.933530] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5574.173695] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5579.576110] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5586.981899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5587.420570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5592.585254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5594.396878] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5603.754768] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5627.364580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5627.365161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5627.381611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5632.495644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5632.503331] Lustre: Skipped 2 previous similar messages [ 5632.952537] Lustre: server umount lustre-OST0000 complete [ 5637.602688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5637.611150] Lustre: Skipped 1 previous similar message [ 5637.616779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5637.623044] Lustre: Skipped 1 previous similar message [ 5642.722049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5642.738039] Lustre: Skipped 1 previous similar message [ 5643.961516] Lustre: server umount lustre-MDT0000 complete [ 5647.848684] LustreError: 184527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5647.877595] LustreError: 184527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5648.122182] LustreError: 184510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140422 with bad export cookie 7182820493280367195 [ 5648.133495] LustreError: 184510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5648.137130] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5648.751648] Lustre: server umount lustre-MDT0001 complete [ 5658.952531] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 5661.391932] Key type lgssc unregistered [ 5661.647857] LNet: 188094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5661.651694] LNetError: 188094:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5661.664907] LNet: Removed LNI 192.168.202.103@tcp [ 5662.382520] Key type .llcrypt unregistered [ 5662.384720] Key type ._llcrypt unregistered [ 5683.497249] Key type ._llcrypt registered [ 5683.499176] Key type .llcrypt registered [ 5683.588734] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5684.474994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5684.503363] alg: No test for adler32 (adler32-zlib) [ 5685.649725] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 5685.833657] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 5687.479193] Key type lgssc registered [ 5688.405768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5695.545713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5695.570905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5697.056389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5701.520959] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5707.235123] LustreError: 189509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5707.255689] LustreError: 189509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5708.938711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5709.215090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5712.858497] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5720.103315] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5725.325760] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5731.449500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5731.854966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5737.838941] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5742.205214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5745.191812] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5752.282593] Lustre: lustre-OST0000: Not available for connect from 192.168.202.3@tcp (stopping) [ 5752.821697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5752.823267] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5752.828851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5752.857046] Lustre: Skipped 1 previous similar message [ 5757.353715] Lustre: server umount lustre-OST0000 complete [ 5764.676291] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8e3683bc5800.ost_server_uuid 50 [ 5765.906584] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5780.966782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5780.973811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5780.986976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5780.994040] Lustre: Skipped 1 previous similar message [ 5783.519956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5783.520282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5785.304446] Lustre: server umount lustre-MDT0000 complete [ 5788.599432] LustreError: 189488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140562 with bad export cookie 552484531874992254 [ 5788.605088] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5788.614916] LustreError: 189488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5788.872491] Lustre: server umount lustre-MDT0001 complete [ 5796.309024] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 00:49:29 (1783140569) [ 5801.604802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5802.056114] LustreError: 193420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5802.145562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5806.057527] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5807.595673] LustreError: 193421:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5812.712901] LustreError: 193420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5813.637854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5813.983783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5818.635451] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5825.382684] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5830.447915] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5836.585896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5836.928698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5843.051971] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5847.221610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5850.042912] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5855.966273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5856.138776] Lustre: lustre-OST0001: new disk, initializing [ 5856.143969] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5856.149154] Lustre: 196259:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5856.217151] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5861.414597] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5862.543422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5862.569914] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5862.621133] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5868.940514] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5873.126105] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5873.130256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5873.142113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5873.145899] Lustre: Skipped 1 previous similar message [ 5873.156363] Lustre: Skipped 2 previous similar messages [ 5877.252686] Lustre: server umount lustre-OST0000 complete [ 5878.248955] LustreError: 195242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5883.169578] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5883.330578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5887.442803] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5887.613524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5888.491675] LustreError: 196692:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5888.513164] LustreError: 196692:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 5903.856918] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5903.860559] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5903.864570] LustreError: Skipped 1 previous similar message [ 5903.867831] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5903.867841] Lustre: Skipped 1 previous similar message [ 5903.890412] Lustre: Skipped 1 previous similar message [ 5907.472834] Lustre: server umount lustre-OST0001 complete [ 5911.520299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5914.084467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5914.100659] Lustre: Skipped 1 previous similar message [ 5924.320607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5924.326173] Lustre: Skipped 7 previous similar messages [ 5925.343600] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5927.535958] Lustre: server umount lustre-MDT0000 complete [ 5929.440430] LustreError: 196023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5929.478618] LustreError: 196023:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5931.273571] LustreError: 194153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140705 with bad export cookie 552484531874993325 [ 5931.283964] LustreError: 194153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5931.291963] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5931.554605] Lustre: server umount lustre-MDT0001 complete [ 5952.592696] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 00:52:05 (1783140725) [ 5959.197842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5959.526615] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5959.553304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5959.807946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5963.779213] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5971.141191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5971.285266] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5971.357198] Lustre: 199698:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5975.314577] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 5981.570633] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5987.647501] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5993.427617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5993.619919] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5999.681550] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6003.313474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6006.291395] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6011.734973] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6011.829803] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6017.276755] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6019.777838] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6023.295567] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6028.769945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6028.780538] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6028.785135] Lustre: lustre-OST0000: Not available for connect from 192.168.202.3@tcp (stopping) [ 6028.801356] Lustre: Skipped 1 previous similar message [ 6031.507541] Lustre: server umount lustre-OST0000 complete [ 6033.859750] LustreError: 201546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6050.794604] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6050.802282] LustreError: Skipped 2 previous similar messages [ 6052.364437] Lustre: server umount lustre-OST0001 complete [ 6070.240266] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6070.456233] Lustre: server umount lustre-MDT0000 complete [ 6074.327437] LustreError: 199689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140848 with bad export cookie 552484531874994732 [ 6074.333062] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6074.339378] LustreError: 199689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6074.627779] Lustre: server umount lustre-MDT0001 complete [ 6096.896112] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 00:54:29 (1783140869) [ 6111.058297] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 6118.858134] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 6151.315235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6151.479757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6151.498433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6151.568455] Lustre: lustre-MDT0000: new disk, initializing [ 6151.630147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6151.634141] Lustre: Skipped 3 previous similar messages [ 6151.646217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6155.277544] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6164.378930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6164.563768] Lustre: 207786:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6164.652749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6164.656105] Lustre: Skipped 1 previous similar message [ 6164.758935] Lustre: lustre-MDT0001: new disk, initializing [ 6164.933366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6164.958252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6169.184519] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6177.593936] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6182.710210] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6189.124758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6189.372258] Lustre: lustre-OST0000: new disk, initializing [ 6189.380037] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6189.390116] Lustre: 209757:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6195.272735] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6196.793379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6196.804633] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6196.937849] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6204.256294] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6208.204543] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6208.428974] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6213.102684] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6213.303313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6217.698125] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6217.704847] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6217.712751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6217.725491] Lustre: Skipped 6 previous similar messages [ 6217.740260] Lustre: Skipped 12 previous similar messages [ 6221.400283] Lustre: server umount lustre-OST0000 complete [ 6230.829823] Lustre: server umount lustre-MDT0000 complete [ 6233.076074] LustreError: 207798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6233.107239] LustreError: 207798:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 6234.858111] LustreError: 208581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141008 with bad export cookie 552484531874996517 [ 6234.866267] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6234.870128] LustreError: 208581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6235.183255] Lustre: server umount lustre-MDT0001 complete [ 6242.530551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6247.237317] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6255.105092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6259.018729] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6265.371411] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6270.254817] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6276.056918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6281.456252] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6288.093684] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6291.081913] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6291.269271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6294.727774] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6294.914305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6298.081918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6298.111393] Lustre: Skipped 3 previous similar messages [ 6302.594491] Lustre: server umount lustre-OST0000 complete [ 6307.764672] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6311.365752] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6339.294243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6339.563198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6339.568148] Lustre: Skipped 5 previous similar messages [ 6344.619202] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6350.700122] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6358.638255] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 6 sec [ 6363.438896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6363.443828] Lustre: Skipped 5 previous similar messages [ 6363.447942] LustreError: 216060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6363.466317] LustreError: 216060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6363.609466] Lustre: server umount lustre-OST0000 complete [ 6368.229401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6368.245741] LustreError: Skipped 2 previous similar messages [ 6373.272040] Lustre: server umount lustre-MDT0000 complete [ 6376.274028] LustreError: 211951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141150 with bad export cookie 552484531874997399 [ 6376.285847] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6376.295393] LustreError: 211951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6376.607600] Lustre: server umount lustre-MDT0001 complete [ 6383.792527] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 00:59:17 (1783141157) [ 6389.446576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6392.837753] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6398.938291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6402.616594] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6408.519813] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6412.875810] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6418.387816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6424.330237] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6430.709554] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6435.284364] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6435.482811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6439.817357] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6440.018175] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6445.704159] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6445.848067] Lustre: lustre-OST0001: new disk, initializing [ 6445.851492] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6445.854609] Lustre: 221300:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6451.858329] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6452.293330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6452.302602] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6452.372484] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6460.343555] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6465.018765] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6465.233874] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6469.600881] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6469.830465] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6473.207504] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6473.227300] Lustre: Skipped 3 previous similar messages [ 6478.152741] Lustre: server umount lustre-OST0001 complete [ 6484.692690] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6484.958313] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6489.399147] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6489.557222] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6517.105068] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6521.671019] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6522.859596] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6522.871325] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6522.877944] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6527.751847] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6529.169304] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6533.094997] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6533.101814] LustreError: Skipped 2 previous similar messages [ 6536.617425] Lustre: server umount lustre-OST0001 complete [ 6565.344460] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6565.440662] Lustre: server umount lustre-OST0000 complete [ 6574.216176] Lustre: server umount lustre-MDT0000 complete [ 6576.325417] LustreError: 221299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141350 with bad export cookie 552484531874998351 [ 6576.331194] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6576.331404] LustreError: 221299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6582.598226] Lustre: server umount lustre-MDT0001 complete [ 6598.553569] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 01:02:51 (1783141371) [ 6603.166529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6603.380215] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6603.405112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6603.545659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6603.549695] Lustre: Skipped 5 previous similar messages [ 6606.337979] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6611.598209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6611.648639] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6611.661107] Lustre: 226556:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6614.657373] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6619.415426] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6623.363658] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6628.225144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6628.360826] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6632.423580] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6636.951475] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6643.409949] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6643.459969] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6647.418516] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6652.289765] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6653.384788] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6656.586303] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6656.763759] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6659.856820] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6659.988925] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6663.651519] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e36889ac000.ost_server_uuid 50 [ 6664.617888] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8e36889ac000.ost_server_uuid in FULL state after 0 sec [ 6665.307531] Lustre: Permanently deactivating lustre-OST0001 [ 6665.317265] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6667.817215] Lustre: Permanently reactivating lustre-OST0001 [ 6672.556565] Lustre: server umount lustre-OST0001 complete [ 6674.927111] LustreError: 228401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6674.934702] LustreError: 228401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 39 previous similar messages [ 6675.939912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6675.946080] Lustre: Skipped 18 previous similar messages [ 6681.350606] Lustre: server umount lustre-OST0000 complete [ 6690.412674] Lustre: server umount lustre-MDT0000 complete [ 6695.725188] LustreError: 226548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141469 with bad export cookie 552484531874999639 [ 6695.726606] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6695.735536] LustreError: 226548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6696.531634] Lustre: server umount lustre-MDT0001 complete [ 6734.410294] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 01:05:06 (1783141506) [ 6742.824796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6743.162901] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6743.191419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6743.198229] Lustre: Skipped 5 previous similar messages [ 6747.546965] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6756.772669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6756.918635] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6757.004313] Lustre: 233246:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6763.469445] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6773.060406] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6780.512287] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6789.295809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6789.421994] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6789.436441] Lustre: Permanently deactivating lustre-OST0000 [ 6789.445512] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6794.831839] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6796.648602] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6803.361400] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6811.089130] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6816.815421] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6817.424763] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6823.660317] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6825.391629] Lustre: Permanently reactivating lustre-OST0000 [ 6825.421605] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6825.427784] Lustre: Skipped 2 previous similar messages [ 6833.283067] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6833.300427] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 38 seconds. I think it's dead, and I am evicting it. exp 00000000e8687b82, cur 1783141607 deadline 1783141599 last 1783141569 [ 6833.392828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6842.850131] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6842.857973] LustreError: Skipped 8 previous similar messages [ 6842.864194] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6842.880025] Lustre: Skipped 12 previous similar messages [ 6843.155177] Lustre: server umount lustre-OST0001 complete [ 6853.316776] Lustre: server umount lustre-OST0000 complete [ 6863.276799] Lustre: server umount lustre-MDT0000 complete [ 6867.326264] LustreError: 233237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141641 with bad export cookie 552484531875001284 [ 6867.337948] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6867.689835] Lustre: server umount lustre-MDT0001 complete [ 6878.679634] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 6881.817645] Key type lgssc unregistered [ 6882.314780] LNet: 238122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6882.322906] LNetError: 238122:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6882.351463] LNet: Removed LNI 192.168.202.103@tcp [ 6883.610409] Key type .llcrypt unregistered [ 6883.615174] Key type ._llcrypt unregistered [ 6907.221779] Key type ._llcrypt registered [ 6907.225301] Key type .llcrypt registered [ 6907.349381] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 6908.574378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6908.612300] alg: No test for adler32 (adler32-zlib) [ 6909.630648] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 6909.844530] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 6911.495366] Key type lgssc registered [ 6912.832633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6922.788145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6922.811856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6924.319722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6928.532478] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6934.497476] LustreError: 239592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6934.523110] LustreError: 239592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6936.528607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6936.615054] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6936.632725] Lustre: 239582:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6936.637658] Lustre: Permanently deactivating lustre-MDT0001 [ 6936.643674] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6936.914234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6941.387856] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6942.770032] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6948.728966] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6955.968388] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6963.020805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6963.319209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6969.448577] LustreError: 241434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6969.479864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6969.752448] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6974.961037] LustreError: 241435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6977.024304] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6980.069555] LustreError: 241436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6981.127735] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6984.546846] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6984.741680] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6989.837928] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6989.846861] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6990.691774] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 6998.137091] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7002.875232] Lustre: Permanently reactivating lustre-MDT0001 [ 7002.924755] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7002.931775] Lustre: Skipped 2 previous similar messages [ 7004.581809] Lustre: lustre-MDT0001: Client 98f53eb0-cce3-4179-8afe-7a2f2ec271e4 (at 192.168.202.3@tcp) reconnecting [ 7011.256674] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7011.984339] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7011.993099] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7011.999548] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7012.007678] LustreError: 243354:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7012.009864] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7012.018167] LustreError: 243354:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7012.018296] LustreError: 243354:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.103@tcp: cfg command failed: rc = -108 [ 7012.018319] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7012.018319] [ 7012.021103] LustreError: 239584:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7012.972793] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7017.736328] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7017.941934] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7023.677496] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7023.892107] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7026.822072] Lustre: Permanently deactivating lustre-MDT0001 [ 7026.838778] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7026.843679] Lustre: Skipped 2 previous similar messages [ 7037.039177] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7044.576511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7044.580423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7044.590275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7046.116171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7046.120159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7046.129029] Lustre: Skipped 2 previous similar messages [ 7046.135662] Lustre: Skipped 3 previous similar messages [ 7050.147110] Lustre: server umount lustre-MDT0000 complete [ 7051.236453] LustreError: 240337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7051.272148] LustreError: 240337:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7053.979711] LustreError: 239573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141828 with bad export cookie 4420672557215496697 [ 7053.990372] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7053.995130] LustreError: 239573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7054.250254] Lustre: server umount lustre-MDT0001 complete [ 7063.834207] Lustre: server umount lustre-OST0000 complete [ 7073.880089] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 01:10:47 (1783141847) [ 7083.174424] Lustre: server umount lustre-OST0001 complete [ 7089.426172] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 7098.217823] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7131.203449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7131.398977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7131.410666] Lustre: Skipped 2 previous similar messages [ 7131.428670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7131.492952] Lustre: lustre-MDT0000: new disk, initializing [ 7131.605326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7131.627260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7136.322932] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7145.989592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7146.131627] Lustre: 247900:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7146.181602] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7146.185539] Lustre: Skipped 1 previous similar message [ 7146.289552] Lustre: lustre-MDT0001: new disk, initializing [ 7146.370819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7146.412181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7146.426029] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7150.530629] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7159.362676] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7164.372453] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7170.766689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7171.067986] Lustre: lustre-OST0000: new disk, initializing [ 7171.074075] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7171.082970] Lustre: 249866:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7171.171696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7177.276390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7177.292464] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7177.432075] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7177.479639] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7186.947191] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7189.743114] LustreError: 247907:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7198.427731] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7198.612923] Lustre: lustre-OST0001: new disk, initializing [ 7198.616632] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7198.620230] Lustre: 250981:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7199.784200] LustreError: 247907:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7199.806255] LustreError: 247906:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7204.942559] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7205.469668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7205.482497] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7205.611968] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7209.847149] LustreError: 247906:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7214.244122] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7215.936928] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7221.217828] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7221.229187] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7221.257892] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7221.266296] Lustre: Skipped 1 previous similar message [ 7226.380111] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7226.394764] Lustre: Skipped 2 previous similar messages [ 7231.459736] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7231.468681] Lustre: Skipped 2 previous similar messages [ 7231.967135] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7232.073830] Lustre: server umount lustre-OST0001 complete [ 7232.475068] LustreError: 249851:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7249.377886] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7249.380641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7249.387256] LustreError: Skipped 2 previous similar messages [ 7249.389835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7249.405214] Lustre: Skipped 2 previous similar messages [ 7249.439787] Lustre: Skipped 1 previous similar message [ 7251.945937] LustreError: 249851:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7251.973703] LustreError: 249851:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 7254.730989] Lustre: server umount lustre-OST0000 complete [ 7259.109336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7259.117724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7264.340969] Lustre: server umount lustre-MDT0000 complete [ 7267.870388] LustreError: 247891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142041 with bad export cookie 4420672557215499511 [ 7267.884368] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7267.900611] LustreError: 247891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7274.491238] Lustre: server umount lustre-MDT0001 complete [ 7285.008707] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 7287.792450] Key type lgssc unregistered [ 7288.058237] LNet: 253017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7288.071785] LNetError: 253017:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7288.092977] LNet: Removed LNI 192.168.202.103@tcp [ 7288.733797] Key type .llcrypt unregistered [ 7288.736088] Key type ._llcrypt unregistered [ 7323.112337] Key type ._llcrypt registered [ 7323.116356] Key type .llcrypt registered [ 7323.191579] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7324.306313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7324.363105] alg: No test for adler32 (adler32-zlib) [ 7325.399052] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 7325.632935] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 7327.279183] Key type lgssc registered [ 7328.249927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7335.128962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7335.143969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7336.400923] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7336.430886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7336.589603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7340.322288] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7350.804922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7350.887958] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7350.920544] Lustre: 255227:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7351.176158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7355.960261] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7363.649300] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7368.797665] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7374.655979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7374.821467] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7375.052069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7380.275740] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7384.709475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7387.177807] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7435.747966] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7435.757646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7435.772668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7436.263891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7436.269735] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7436.282729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7436.286253] Lustre: Skipped 1 previous similar message [ 7441.384963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7441.389770] Lustre: Skipped 1 previous similar message [ 7441.863414] Lustre: server umount lustre-OST0000 complete [ 7445.756739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7458.094604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7458.573903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7465.553839] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7466.732364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7466.750877] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7471.912096] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7471.928401] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7472.551050] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7513.059468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7513.076592] Lustre: Skipped 1 previous similar message [ 7513.077200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7516.918150] Lustre: server umount lustre-OST0000 complete [ 7523.296630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7523.307269] Lustre: Skipped 1 previous similar message [ 7523.310327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7523.327220] Lustre: Skipped 2 previous similar messages [ 7526.698218] Lustre: server umount lustre-MDT0000 complete [ 7529.837861] LustreError: 255220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142303 with bad export cookie 13510093858432363902 [ 7529.843470] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7529.858252] LustreError: 255220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7530.100387] Lustre: server umount lustre-MDT0001 complete [ 7538.262916] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 7540.768699] Key type lgssc unregistered [ 7540.979740] LNet: 260646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7540.984735] LNetError: 260646:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7541.007228] LNet: Removed LNI 192.168.202.103@tcp [ 7541.955484] Key type .llcrypt unregistered [ 7541.958427] Key type ._llcrypt unregistered [ 7563.051813] Key type ._llcrypt registered [ 7563.055185] Key type .llcrypt registered [ 7563.170635] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7564.134082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7564.171195] alg: No test for adler32 (adler32-zlib) [ 7565.178179] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 7565.386960] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 7567.055230] Key type lgssc registered [ 7568.312126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7575.771561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7575.787559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7577.129476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7580.736513] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7587.299306] LustreError: 262043:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7587.309685] LustreError: 262043:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7588.057779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7588.299132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7592.108991] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7599.243584] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7603.858832] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7611.459868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7611.880988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7617.989239] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7622.155301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7626.012484] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7649.767888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7649.777330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7649.797569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7652.833164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7652.836969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7652.846513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7652.873596] Lustre: Skipped 1 previous similar message [ 7655.864735] Lustre: server umount lustre-OST0000 complete [ 7663.078472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7663.088938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7663.092210] Lustre: Skipped 1 previous similar message [ 7663.107053] Lustre: Skipped 1 previous similar message [ 7666.738823] Lustre: server umount lustre-MDT0000 complete [ 7668.194696] LustreError: 263676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7668.210042] LustreError: 263676:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7669.875736] LustreError: 262025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142443 with bad export cookie 2969444449079442968 [ 7669.881556] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7669.892178] LustreError: 262025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7670.437474] Lustre: server umount lustre-MDT0001 complete [ 7681.707878] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 7685.300741] Key type lgssc unregistered [ 7685.642186] LNet: 266350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7685.646565] LNetError: 266350:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7686.694268] LNet: Removed LNI 192.168.202.103@tcp [ 7687.623861] Key type .llcrypt unregistered [ 7687.625962] Key type ._llcrypt unregistered [ 7704.597884] Key type ._llcrypt registered [ 7704.600189] Key type .llcrypt registered [ 7704.719985] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7706.084283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7706.208772] alg: No test for adler32 (adler32-zlib) [ 7707.238718] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 7707.596734] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 7709.343364] Key type lgssc registered [ 7710.523976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7717.873387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7717.898479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7719.491589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7723.990767] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7729.640434] LustreError: 267571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7729.660745] LustreError: 267571:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7732.597292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7733.004990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7737.415894] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7744.098064] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7748.589874] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7755.132449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7755.384832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7761.447308] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7764.600734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7767.735162] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7779.303519] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7779.308907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7779.321589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7780.320507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7780.327384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7780.337439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7780.367984] Lustre: Skipped 1 previous similar message [ 7781.900338] Lustre: server umount lustre-OST0000 complete [ 7785.441778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7785.447541] Lustre: Skipped 1 previous similar message [ 7785.450313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7790.567773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7790.573952] Lustre: Skipped 2 previous similar messages [ 7791.467692] Lustre: server umount lustre-MDT0000 complete [ 7794.809328] LustreError: 267554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142568 with bad export cookie 16838814024546547198 [ 7794.816201] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7794.819161] LustreError: 267554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7795.199704] Lustre: server umount lustre-MDT0001 complete [ 7804.671438] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 7806.959390] Key type lgssc unregistered [ 7807.172807] LNet: 271249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7807.179020] LNetError: 271249:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7807.197834] LNet: Removed LNI 192.168.202.103@tcp [ 7808.050617] Key type .llcrypt unregistered [ 7808.052183] Key type ._llcrypt unregistered [ 7826.722177] Key type ._llcrypt registered [ 7826.724590] Key type .llcrypt registered [ 7826.803599] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7827.465686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7827.586991] alg: No test for adler32 (adler32-zlib) [ 7828.607335] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 7828.836238] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 7830.527562] Key type lgssc registered [ 7831.520820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7837.422579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7837.452781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7838.858623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7842.404953] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7848.418415] LustreError: 272644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7848.447153] LustreError: 272644:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7849.889949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7850.147519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7853.634256] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7860.372185] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7864.639193] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7870.019638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7870.410775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7876.580881] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7882.496132] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7883.767407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7906.274982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7906.288156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7906.300051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7909.352521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7909.357662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7909.358241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7911.679299] Lustre: server umount lustre-OST0000 complete [ 7917.024382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7917.030499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7917.047390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7917.057213] Lustre: Skipped 1 previous similar message [ 7919.590294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7919.591810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7919.611863] Lustre: Skipped 1 previous similar message [ 7924.704491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7924.716233] Lustre: Skipped 1 previous similar message [ 7929.311314] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7929.473218] Lustre: server umount lustre-MDT0000 complete [ 7929.825395] LustreError: 273373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7929.845396] LustreError: 273373:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7933.383380] LustreError: 272625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142707 with bad export cookie 10598181880544358919 [ 7933.392975] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7933.404917] LustreError: 272625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7933.745888] Lustre: server umount lustre-MDT0001 complete [ 7943.359113] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 7945.641209] Key type lgssc unregistered [ 7945.904640] LNet: 276950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7945.921454] LNetError: 276950:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7946.986847] LNet: Removed LNI 192.168.202.103@tcp [ 7948.000529] Key type .llcrypt unregistered [ 7948.002846] Key type ._llcrypt unregistered [ 7963.101667] Key type ._llcrypt registered [ 7963.103656] Key type .llcrypt registered [ 7963.194199] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7963.842098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7963.866049] alg: No test for adler32 (adler32-zlib) [ 7964.882911] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 7965.133290] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 7966.783289] Key type lgssc registered [ 7967.857980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7974.573836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7974.594815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7976.054574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7979.322546] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7986.471751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7986.660533] LustreError: 278183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7986.678910] LustreError: 278183:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7986.931501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7990.466144] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 7996.414800] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8000.921509] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8007.040985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8007.290423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8012.199273] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8018.040674] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8021.619895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8032.235151] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8032.246028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8032.262129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8032.269411] Lustre: Skipped 1 previous similar message [ 8037.313061] Lustre: server umount lustre-OST0000 complete [ 8042.466137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8042.470823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8042.482474] Lustre: Skipped 2 previous similar messages [ 8046.435281] Lustre: server umount lustre-MDT0000 complete [ 8047.588851] LustreError: 278194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8047.599846] LustreError: 278194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8049.374721] LustreError: 278148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142823 with bad export cookie 10758589970476330415 [ 8049.383908] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8049.384751] LustreError: 278148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8049.700919] Lustre: server umount lustre-MDT0001 complete [ 8058.395829] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 8060.651455] Key type lgssc unregistered [ 8060.883440] LNet: 281882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8060.893400] LNetError: 281882:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8060.909131] LNet: Removed LNI 192.168.202.103@tcp [ 8061.647753] Key type .llcrypt unregistered [ 8061.649716] Key type ._llcrypt unregistered [ 8072.155522] Key type ._llcrypt registered [ 8072.158402] Key type .llcrypt registered [ 8072.239553] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8085.369925] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 8097.416555] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 8098.370123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8098.421244] alg: No test for adler32 (adler32-zlib) [ 8099.385605] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 8099.560397] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 8101.199753] Key type lgssc registered [ 8102.050716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8128.072467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8128.097971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8129.364367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8129.390625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8129.445646] Lustre: lustre-MDT0000: new disk, initializing [ 8129.543558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8129.559198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8133.354021] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8141.341345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8141.474339] Lustre: 285410:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8141.523759] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8141.527837] Lustre: Skipped 1 previous similar message [ 8141.605557] Lustre: lustre-MDT0001: new disk, initializing [ 8141.687057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8141.740267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8141.747716] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8145.070288] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8152.760592] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8157.370559] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8162.891748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8163.038934] Lustre: lustre-OST0000: new disk, initializing [ 8163.041835] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8163.046799] Lustre: 287378:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8163.112331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8167.408371] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8172.616337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8172.627550] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8172.742817] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8174.358065] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8178.080660] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8178.242264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8181.716285] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8181.852704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8183.264050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8183.266966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8183.271283] LustreError: Skipped 1 previous similar message [ 8183.286438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8183.292188] Lustre: Skipped 1 previous similar message [ 8188.390177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8189.524247] Lustre: server umount lustre-OST0000 complete [ 8192.992315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8192.996976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8193.006020] Lustre: Skipped 1 previous similar message [ 8193.014809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8193.021022] Lustre: Skipped 1 previous similar message [ 8198.624648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8198.635834] Lustre: Skipped 3 previous similar messages [ 8198.847739] Lustre: server umount lustre-MDT0000 complete [ 8201.838735] LustreError: 285402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142975 with bad export cookie 8183608319854324690 [ 8201.847118] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8202.163493] Lustre: server umount lustre-MDT0001 complete [ 8208.518612] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 01:29:41 (1783142981) [ 8213.463966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8213.885172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8217.106472] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8219.104565] LustreError: 289789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8219.117613] LustreError: 289789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8223.201445] LustreError: 289788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8223.247057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8223.534092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8226.995348] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8232.776484] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8237.183539] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8242.501442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8242.772711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8246.864543] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8252.003990] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8268.770374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8268.782597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8268.795150] Lustre: Skipped 1 previous similar message [ 8268.800531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8268.807880] Lustre: Skipped 1 previous similar message [ 8270.144640] Lustre: server umount lustre-OST0000 complete [ 8272.740922] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8272.745186] LustreError: Skipped 1 previous similar message [ 8272.747638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8272.757627] Lustre: Skipped 1 previous similar message [ 8278.905589] Lustre: server umount lustre-MDT0000 complete [ 8279.010839] LustreError: 289784:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8279.022744] LustreError: 289784:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8281.174904] LustreError: 289768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143055 with bad export cookie 8183608319854325572 [ 8281.181941] LustreError: 290525:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 8281.182222] LustreError: 289768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8281.183182] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8281.424707] Lustre: server umount lustre-MDT0001 complete [ 8288.844956] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 8290.673532] Key type lgssc unregistered [ 8290.911784] LNet: 293343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8290.923174] LNetError: 293343:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8290.944945] LNet: Removed LNI 192.168.202.103@tcp [ 8291.508617] Key type .llcrypt unregistered [ 8291.512842] Key type ._llcrypt unregistered [ 8308.980789] Key type ._llcrypt registered [ 8308.982824] Key type .llcrypt registered [ 8309.065910] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 8319.934902] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 8320.435772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8320.479241] alg: No test for adler32 (adler32-zlib) [ 8321.433075] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 8321.590686] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 8323.199174] Key type lgssc registered [ 8323.933650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8352.084972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8352.120978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8353.278607] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8353.297794] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8353.346922] Lustre: lustre-MDT0000: new disk, initializing [ 8353.390861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8353.404925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8356.360870] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8363.927546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8364.028543] Lustre: 297145:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8364.080536] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8364.083843] Lustre: Skipped 1 previous similar message [ 8364.156724] Lustre: lustre-MDT0001: new disk, initializing [ 8364.240612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8364.270211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8364.280786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8367.248950] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8373.717331] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8377.263870] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8381.735520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8381.994379] Lustre: lustre-OST2710: new disk, initializing [ 8381.998761] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8382.011689] Lustre: 299113:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8382.094129] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8386.373743] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8391.708491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8391.713575] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8391.776980] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8392.474079] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8396.525246] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8396.643307] Lustre: lustre-OST03e8: new disk, initializing [ 8396.645798] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8396.650317] Lustre: 300175:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8396.706677] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8398.117485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8398.125518] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8398.196248] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8400.513574] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8405.545886] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8411.443698] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8411.644673] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8414.619691] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8414.801112] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8417.742767] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8417.887476] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8420.734323] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8420.856454] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8423.913521] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8423.916413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8423.923427] Lustre: Skipped 1 previous similar message [ 8423.933343] Lustre: Skipped 2 previous similar messages [ 8429.029865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8429.033426] Lustre: Skipped 1 previous similar message [ 8429.608507] Lustre: server umount lustre-MDT0000 complete [ 8431.762346] LustreError: 297140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143205 with bad export cookie 10153666598094959542 [ 8431.775653] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8431.777195] LustreError: 297140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8432.049788] Lustre: server umount lustre-MDT0001 complete [ 8444.886959] Lustre: server umount lustre-OST2710 complete [ 8457.479303] Lustre: server umount lustre-OST03e8 complete [ 8467.888584] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 8473.242632] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 8494.723527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8494.832933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8494.847981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8494.914278] Lustre: lustre-MDT0000: new disk, initializing [ 8494.954670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8494.976785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8497.873642] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8505.120372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8505.207560] Lustre: 305414:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8505.244528] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8505.246723] Lustre: Skipped 1 previous similar message [ 8505.291636] Lustre: lustre-MDT0001: new disk, initializing [ 8505.361125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8505.386444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8505.390456] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8508.358352] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8514.582820] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8517.960320] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8522.320099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8522.493210] Lustre: lustre-OST0000: new disk, initializing [ 8522.496421] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8522.501562] Lustre: 307378:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8522.560478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8526.415387] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8528.931542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8528.937087] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8528.993607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8532.922718] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8535.825986] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8535.959834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8538.494132] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8538.634985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8544.232073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8544.241750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8544.243019] LustreError: Skipped 1 previous similar message [ 8544.246559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8544.246568] Lustre: Skipped 2 previous similar messages [ 8544.252555] Lustre: Skipped 3 previous similar messages [ 8545.930634] Lustre: server umount lustre-OST0000 complete [ 8549.347264] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8549.350437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8549.361659] Lustre: Skipped 1 previous similar message [ 8549.368783] Lustre: Skipped 2 previous similar messages [ 8554.465549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8554.471531] Lustre: Skipped 1 previous similar message [ 8554.583925] Lustre: server umount lustre-MDT0000 complete [ 8556.871042] LustreError: 305406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143330 with bad export cookie 10153666598094961222 [ 8556.877355] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8556.878154] LustreError: 305406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8557.093701] Lustre: server umount lustre-MDT0001 complete [ 8562.794859] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 01:35:36 (1783143336) [ 8563.808831] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8564.982800] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 01:35:38 (1783143338) [ 8567.252704] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 8585.489473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8585.681661] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8585.701721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8585.832855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8588.541212] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8594.062808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8594.135040] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8594.156547] Lustre: 311232:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8597.015805] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8615.717918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8615.861180] Lustre: Denying initial registration attempt from nid 192.168.202.103@tcp, specified as failover [ 8615.866790] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8615.871195] LustreError: 312689:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8615.875959] LustreError: 312689:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8615.879936] LustreError: 312689:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8615.917074] Lustre: server umount lustre-OST0000 complete [ 8615.919647] LustreError: 312689:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8623.073073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8623.078938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8623.084753] Lustre: Skipped 1 previous similar message [ 8626.459065] Lustre: server umount lustre-MDT0000 complete [ 8628.994941] LustreError: 311226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143403 with bad export cookie 10153666598094962097 [ 8628.997814] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8629.006513] LustreError: 311226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8629.243586] Lustre: server umount lustre-MDT0001 complete [ 8636.136463] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 8637.804386] Key type lgssc unregistered [ 8637.982183] LNet: 313807:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8637.989262] LNetError: 313807:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8638.005489] LNet: Removed LNI 192.168.202.103@tcp [ 8638.435712] Key type .llcrypt unregistered [ 8638.437200] Key type ._llcrypt unregistered [ 8645.606959] Key type ._llcrypt registered [ 8645.610444] Key type .llcrypt registered [ 8645.676967] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 8646.340910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8646.440206] alg: No test for adler32 (adler32-zlib) [ 8647.428520] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 8647.590822] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 8649.215155] Key type lgssc registered [ 8649.930519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8682.331446] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 8686.675088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8686.689283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8687.811599] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8687.826265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8687.914599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8689.937417] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8694.481682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8694.527333] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8694.544445] Lustre: 316550:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8694.690128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8697.077824] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8700.693102] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8703.548442] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8720.552115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8720.652506] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8720.766737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8723.753899] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8727.468996] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8728.728793] Lustre: server umount lustre-OST0000 complete [ 8735.203578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8735.214236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8737.178173] Lustre: server umount lustre-MDT0000 complete [ 8739.435448] LustreError: 316542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143513 with bad export cookie 12614628651481549320 [ 8739.443763] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8739.446781] LustreError: 316542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8739.683091] Lustre: server umount lustre-MDT0001 complete [ 8745.771420] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 8747.282880] Key type lgssc unregistered [ 8747.457765] LNet: 320120:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8747.462711] LNetError: 320120:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8747.473867] LNet: Removed LNI 192.168.202.103@tcp [ 8747.947066] Key type .llcrypt unregistered [ 8747.948932] Key type ._llcrypt unregistered [ 8761.141411] Key type ._llcrypt registered [ 8761.142573] Key type .llcrypt registered [ 8761.212886] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 8761.799769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8761.834218] alg: No test for adler32 (adler32-zlib) [ 8762.735125] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 8762.877202] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 8764.504232] Key type lgssc registered [ 8765.168519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8769.798893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8769.818907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8771.076153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8773.597717] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8778.369422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8778.587501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8780.762206] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8784.705254] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8787.839085] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8793.648026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8793.849814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8796.987311] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8800.456655] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8812.997553] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.3@tcp (stopping) [ 8813.537427] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8813.540897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8813.547459] Lustre: Skipped 1 previous similar message [ 8813.555024] Lustre: Skipped 2 previous similar messages [ 8817.337737] Lustre: server umount lustre-MDT0000 complete [ 8818.107138] LustreError: 321509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8818.114784] LustreError: 321509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8818.657313] LustreError: 321510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8818.666375] LustreError: 321510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8819.238463] LustreError: 321494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143593 with bad export cookie 989214289184095499 [ 8819.240311] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8819.247897] LustreError: 321494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8823.224693] LustreError: 321511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8823.236662] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.3@tcp (stopping) [ 8823.781192] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8823.792964] Lustre: Skipped 1 previous similar message [ 8825.493907] Lustre: server umount lustre-MDT0001 complete [ 8828.968558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8833.598787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8845.285389] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xdba6527a98f190b [ 8845.298368] Lustre: MGC192.168.202.103@tcp: Connection restored to 0@lo (at 0@lo) [ 8845.455500] LustreError: 323343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8845.465306] LustreError: 323343:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8845.501905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8845.536297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 8847.696923] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8850.663082] LustreError: 323344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8850.679598] LustreError: 323344:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8852.424861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8852.629435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8854.719114] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8855.723105] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8858.679771] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8859.546279] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8862.688660] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8864.740303] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 1 sec [ 8868.320925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8868.326295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8868.334230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8868.338883] Lustre: Skipped 2 previous similar messages [ 8871.396791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8871.403812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8876.513806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8876.517030] Lustre: Skipped 2 previous similar messages [ 8881.119143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8881.199030] Lustre: server umount lustre-OST0000 complete [ 8883.681332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8883.684481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8886.755618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8886.763442] Lustre: Skipped 3 previous similar messages [ 8889.368632] Lustre: server umount lustre-MDT0000 complete [ 8891.442518] LustreError: 321494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143665 with bad export cookie 989214289184102667 [ 8891.446583] LustreError: 321494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8891.446693] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8891.639230] Lustre: server umount lustre-MDT0001 complete [ 8897.025117] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 8898.542809] Key type lgssc unregistered [ 8898.716624] LNet: 327775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8898.724565] LNetError: 327775:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8899.753132] LNet: Removed LNI 192.168.202.103@tcp [ 8900.336488] Key type .llcrypt unregistered [ 8900.337797] Key type ._llcrypt unregistered [ 8915.821918] Key type ._llcrypt registered [ 8915.823309] Key type .llcrypt registered [ 8915.887627] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 8916.477686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8916.501203] alg: No test for adler32 (adler32-zlib) [ 8917.432775] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 8917.637264] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 8919.255172] Key type lgssc registered [ 8919.980940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8925.192420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8925.218488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8926.457812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8928.741104] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8933.247162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8933.435494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8935.611793] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8940.012360] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8943.249664] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8948.705099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8948.712106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8948.721922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8949.216077] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8949.229157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8949.233955] Lustre: Skipped 1 previous similar message [ 8951.050347] Lustre: server umount lustre-MDT0000 complete [ 8952.937272] LustreError: 329152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143727 with bad export cookie 11823480266754815828 [ 8952.938450] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8952.949183] LustreError: 329152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8953.212568] Lustre: server umount lustre-MDT0001 complete [ 8960.940220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8961.091411] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8961.110604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8961.208476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8963.618563] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8968.380168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8968.429398] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8968.449862] Lustre: 331753:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8968.598826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8970.769480] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8974.664091] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8977.170117] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8983.076593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8983.180811] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8983.186506] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8983.189922] LustreError: 333631:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 8983.192635] LustreError: 333631:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8983.196357] LustreError: 333631:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8983.228335] Lustre: server umount lustre-OST0000 complete [ 8983.230478] LustreError: 333631:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8989.348380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8989.456587] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8989.571986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8992.539169] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 8995.866521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 8996.427734] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9000.451032] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9000.512104] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9000.555793] Lustre: lustre-OST0001: new disk, initializing [ 9000.558245] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9000.561928] Lustre: 335174:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9000.618874] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9002.638306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9002.645964] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9002.681485] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9003.925885] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9008.105293] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9013.222846] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9013.223730] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9013.234411] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9013.248168] Lustre: Skipped 1 previous similar message [ 9013.257499] Lustre: Skipped 1 previous similar message [ 9015.716122] Lustre: server umount lustre-OST0001 complete [ 9018.344803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9018.353118] LustreError: 334098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9018.355670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9018.362196] LustreError: 334098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9018.369791] Lustre: Skipped 1 previous similar message [ 9023.459874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9023.461699] LustreError: 334730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9023.464312] Lustre: Skipped 1 previous similar message [ 9023.482751] LustreError: 334730:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9024.092696] Lustre: server umount lustre-OST0000 complete [ 9028.583111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9028.596065] Lustre: Skipped 2 previous similar messages [ 9032.298462] Lustre: server umount lustre-MDT0000 complete [ 9033.698137] LustreError: 331763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9033.704867] LustreError: 331763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9034.377055] LustreError: 331743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143808 with bad export cookie 11823480266754816367 [ 9034.380761] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9034.383587] LustreError: 331743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9034.560915] Lustre: server umount lustre-MDT0001 complete [ 9040.032109] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 9041.651371] Key type lgssc unregistered [ 9041.842112] LNet: 337133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9041.847617] LNetError: 337133:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9041.865181] LNet: Removed LNI 192.168.202.103@tcp [ 9042.405341] Key type .llcrypt unregistered [ 9042.407067] Key type ._llcrypt unregistered [ 9080.773185] Key type ._llcrypt registered [ 9080.777902] Key type .llcrypt registered [ 9080.852365] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 9089.091320] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 9089.757212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9089.771154] alg: No test for adler32 (adler32-zlib) [ 9090.684133] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 9090.790692] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 9092.391187] Key type lgssc registered [ 9092.863057] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9106.780739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9106.787883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9107.911744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9107.926632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9107.972831] Lustre: lustre-MDT0000: new disk, initializing [ 9108.041560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9108.053919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9110.136940] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9116.274789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9116.334615] Lustre: 342035:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9116.371221] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9116.374040] Lustre: Skipped 1 previous similar message [ 9116.433728] Lustre: lustre-MDT0001: new disk, initializing [ 9116.491739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9116.519145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9116.524532] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9118.342148] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9123.377482] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9125.907822] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9129.004778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9129.134499] Lustre: lustre-OST0000: new disk, initializing [ 9129.137131] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9129.140258] Lustre: 344002:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9129.180504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9131.761395] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9135.115906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9135.121513] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9135.174132] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9136.352052] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9138.061575] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9138.142471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9139.674376] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9139.757159] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9145.313227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9145.313789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9145.316607] LustreError: Skipped 1 previous similar message [ 9145.322083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9145.325284] Lustre: Skipped 1 previous similar message [ 9146.589462] Lustre: server umount lustre-OST0000 complete [ 9150.432798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9150.433451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9150.438044] Lustre: Skipped 2 previous similar messages [ 9154.346453] Lustre: server umount lustre-MDT0000 complete [ 9155.946542] LustreError: 342071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143930 with bad export cookie 6692541934786599693 [ 9155.949866] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9155.953039] LustreError: 342071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9156.110623] Lustre: server umount lustre-MDT0001 complete [ 9159.876667] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 01:45:33 (1783143933) [ 9163.163624] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 01:45:36 (1783143936) [ 9167.259674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9167.466373] LustreError: 346753:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9167.473609] LustreError: 346753:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9167.505928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9169.241954] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9172.683673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9172.851242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9174.409487] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9177.239635] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9179.286622] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9181.894273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9182.019175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9184.137592] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9187.148570] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9196.038336] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9196.772940] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9197.459397] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9198.156710] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9199.008738] Lustre: Failing over lustre-MDT0000 [ 9199.169434] Lustre: server umount lustre-MDT0000 complete [ 9202.627820] LustreError: 348751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9202.656704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9202.661058] Lustre: Skipped 2 previous similar messages [ 9207.735609] LustreError: 347481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9207.743123] LustreError: 347481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9212.854069] LustreError: 348751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9212.858563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9212.863881] LustreError: 348751:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9212.930138] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9213.038951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9213.073127] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9214.614235] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9217.974710] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9218.536552] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9218.550676] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9218.567800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9220.565347] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9221.174386] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9223.163593] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9223.946350] Lustre: Failing over lustre-MDT0000 [ 9224.103095] Lustre: server umount lustre-MDT0000 complete [ 9228.224285] LustreError: 348751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9228.232847] LustreError: 348751:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9228.770139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9228.773746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9228.778077] Lustre: Skipped 2 previous similar messages [ 9238.125214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9238.208193] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9238.379967] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9238.461617] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9240.506787] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9243.622880] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9243.625126] Lustre: Skipped 2 previous similar messages [ 9243.637578] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9243.656837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9245.969308] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9246.753941] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9253.856895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9253.856942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9253.858813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9253.858820] Lustre: Skipped 1 previous similar message [ 9253.868024] Lustre: Skipped 1 previous similar message [ 9255.328301] Lustre: server umount lustre-MDT0000 complete [ 9256.738619] LustreError: 346733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144030 with bad export cookie 6692541934786602381 [ 9256.741184] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9256.743822] LustreError: 346733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9256.870449] Lustre: server umount lustre-MDT0001 complete [ 9268.656878] Lustre: server umount lustre-OST0000 complete [ 9275.981025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9276.195450] LustreError: 353082:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9276.204867] LustreError: 353082:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 9276.227660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9276.230740] Lustre: Skipped 1 previous similar message [ 9278.183737] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9282.424045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9284.497469] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9287.811849] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9290.393874] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9293.404058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9296.315760] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9299.638881] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9301.801261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9312.226195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9312.226690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9312.234568] LustreError: Skipped 1 previous similar message [ 9312.235337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9312.247311] Lustre: Skipped 2 previous similar messages [ 9312.258716] Lustre: Skipped 3 previous similar messages [ 9313.970435] Lustre: server umount lustre-OST0000 complete [ 9317.345789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9317.350019] Lustre: Skipped 1 previous similar message [ 9317.592339] Lustre: server umount lustre-MDT0000 complete [ 9319.303912] LustreError: 355681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144093 with bad export cookie 6692541934786603130 [ 9319.307555] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9319.312118] LustreError: 355681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9319.459363] Lustre: server umount lustre-MDT0001 complete [ 9324.119494] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 9325.368743] Key type lgssc unregistered [ 9325.518633] LNet: 356631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9325.524148] LNetError: 356631:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9325.536379] LNet: Removed LNI 192.168.202.103@tcp [ 9326.016180] Key type .llcrypt unregistered [ 9326.017391] Key type ._llcrypt unregistered [ 9338.510771] Key type ._llcrypt registered [ 9338.512530] Key type .llcrypt registered [ 9338.570878] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 9339.015382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9339.047076] alg: No test for adler32 (adler32-zlib) [ 9339.934412] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 9340.035396] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 9341.615156] Key type lgssc registered [ 9342.126745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9345.166509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9345.173151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9346.345987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9348.182799] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9351.501560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9351.652505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9353.622794] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9356.944943] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9359.507933] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9362.612649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9362.820580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9365.454526] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9368.925780] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9371.047781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9387.489576] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9387.495046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9387.496873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9387.504463] Lustre: Skipped 2 previous similar messages [ 9391.384667] Lustre: server umount lustre-MDT0000 complete [ 9392.611857] LustreError: 358121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9392.623084] LustreError: 358121:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9393.175502] LustreError: 358105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144167 with bad export cookie 6829144357548751450 [ 9393.180874] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9393.183715] LustreError: 358105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9393.385748] Lustre: server umount lustre-MDT0001 complete [ 9405.798132] Lustre: server umount lustre-OST0000 complete [ 9411.331828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9411.574315] LustreError: 361854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9411.608526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9413.377246] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9416.672476] LustreError: 361855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9417.046181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9417.229052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9419.107572] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9421.891471] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9423.925318] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9426.645893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9426.809970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9429.251980] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9432.058695] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9436.006762] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9446.369532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9446.370798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9446.375339] Lustre: Skipped 4 previous similar messages [ 9446.378194] Lustre: Skipped 1 previous similar message [ 9450.635752] Lustre: server umount lustre-MDT0000 complete [ 9451.490986] LustreError: 361854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9451.499037] LustreError: 361854:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9452.666718] LustreError: 361834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144226 with bad export cookie 6829144357548752549 [ 9452.671635] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9452.675202] LustreError: 361834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9452.806654] Lustre: server umount lustre-MDT0001 complete [ 9464.859786] Lustre: server umount lustre-OST0000 complete [ 9470.606911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9470.771866] LustreError: 365760:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9470.793718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9472.193208] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9475.263313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9476.788911] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9479.225282] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9481.209839] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9483.947644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9486.426825] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9489.234938] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9495.334108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9505.760159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9505.764801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9505.771079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9505.773909] Lustre: Skipped 2 previous similar messages [ 9507.153871] Lustre: server umount lustre-OST0000 complete [ 9510.880660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9510.881452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9510.886698] Lustre: Skipped 2 previous similar messages [ 9514.713167] Lustre: server umount lustre-MDT0000 complete [ 9516.000163] LustreError: 365779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9516.007188] LustreError: 365779:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9516.078062] LustreError: 367602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144290 with bad export cookie 6829144357548753557 [ 9516.081471] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9516.083372] LustreError: 367602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9516.213508] Lustre: server umount lustre-MDT0001 complete [ 9519.957413] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 9521.099426] Key type lgssc unregistered [ 9521.251581] LNet: 369311:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9521.255627] LNetError: 369311:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9521.268307] LNet: Removed LNI 192.168.202.103@tcp [ 9521.596807] Key type .llcrypt unregistered [ 9521.598500] Key type ._llcrypt unregistered [ 9530.867614] Key type ._llcrypt registered [ 9530.868676] Key type .llcrypt registered [ 9530.903761] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 9531.306911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9531.316463] alg: No test for adler32 (adler32-zlib) [ 9532.177712] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 9532.261050] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 9533.839285] Key type lgssc registered [ 9534.231447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9536.880677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9536.889369] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9536.893244] LustreError: 370728:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9536.905481] LustreError: 370728:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9536.908650] LustreError: 370728:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9536.911627] LustreError: 370728:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9536.914751] LustreError: 370728:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9539.947296] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9539.950137] LustreError: 371124:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9539.959714] LustreError: 371124:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9539.963969] LustreError: 371124:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9539.966707] LustreError: 371124:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9539.969467] LustreError: 371124:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9547.271876] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 9548.311939] Key type lgssc unregistered [ 9548.434271] LNet: 372124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9548.436652] LNetError: 372124:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9548.448268] LNet: Removed LNI 192.168.202.103@tcp [ 9548.761481] Key type .llcrypt unregistered [ 9548.763093] Key type ._llcrypt unregistered [ 9554.784041] Key type ._llcrypt registered [ 9554.785293] Key type .llcrypt registered [ 9554.824525] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 9560.958407] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 9561.379263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9561.386400] alg: No test for adler32 (adler32-zlib) [ 9562.235695] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 9562.323462] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 9563.903156] Key type lgssc registered [ 9564.304639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9575.418264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9575.425155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9576.512072] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9576.524639] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9576.567230] Lustre: lustre-MDT0000: new disk, initializing [ 9576.592156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9576.598612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9577.924602] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9582.077843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9582.129756] Lustre: 375255:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9582.158967] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9582.161196] Lustre: Skipped 1 previous similar message [ 9582.199682] Lustre: lustre-MDT0001: new disk, initializing [ 9582.237832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9582.250710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9582.254736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9583.697414] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9587.471353] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9589.237551] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9591.438572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9591.537943] Lustre: lustre-OST0000: new disk, initializing [ 9591.539801] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9591.542939] Lustre: 377222:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9591.575626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9593.516212] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9594.165359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9594.168994] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9594.201701] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9597.212938] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9598.668248] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9598.739379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9600.162848] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9600.235839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9604.577318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9604.577410] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9604.580206] LustreError: Skipped 1 previous similar message [ 9604.585740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9604.590595] Lustre: Skipped 1 previous similar message [ 9609.698275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9609.700553] Lustre: Skipped 1 previous similar message [ 9614.817042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9614.819733] Lustre: Skipped 1 previous similar message [ 9615.327108] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9615.388403] Lustre: server umount lustre-OST0000 complete [ 9618.400190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9618.407616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9618.411926] Lustre: Skipped 1 previous similar message [ 9618.414919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9619.936759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9625.056553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9625.065442] Lustre: Skipped 3 previous similar messages [ 9631.199122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9631.266991] Lustre: server umount lustre-MDT0000 complete [ 9632.798405] LustreError: 377221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144406 with bad export cookie 11119341355310388757 [ 9632.800228] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9632.805890] LustreError: 377221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9632.963480] Lustre: server umount lustre-MDT0001 complete [ 9636.054866] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9636.642324] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 01:53:30 (1783144410) [ 9639.315587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9639.574495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9641.307277] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9644.808661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9644.953826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9646.580633] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9649.765835] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9651.894451] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9654.754443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9654.868042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9657.104560] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9660.046845] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9662.794449] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9662.864728] Lustre: lustre-OST0001: new disk, initializing [ 9662.866938] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9662.870214] Lustre: 382513:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9665.019445] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9668.845435] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9670.146772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9670.151529] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9670.179515] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9675.200037] Lustre: lustre-OST0001: Not available for connect from 192.168.202.3@tcp (stopping) [ 9675.202788] Lustre: Skipped 2 previous similar messages [ 9675.235967] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9675.236330] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9675.243298] LustreError: Skipped 1 previous similar message [ 9676.801862] Lustre: server umount lustre-OST0001 complete [ 9680.355196] LustreError: 381499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9680.361497] LustreError: 381499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9685.476974] LustreError: 383236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9685.484109] LustreError: 383236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9689.568072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9689.571815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9689.576897] Lustre: Skipped 1 previous similar message [ 9690.593752] LustreError: 381674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9690.600132] LustreError: 381674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9695.714874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9695.715964] LustreError: 381500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9695.717033] Lustre: Skipped 6 previous similar messages [ 9695.721516] LustreError: 381500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9700.835601] LustreError: 381498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9700.842865] LustreError: 381498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9703.391131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9703.466902] Lustre: server umount lustre-OST0000 complete [ 9705.952418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9705.957084] Lustre: Skipped 1 previous similar message [ 9711.325726] Lustre: server umount lustre-MDT0000 complete [ 9712.742119] LustreError: 379660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144486 with bad export cookie 11119341355310389625 [ 9712.745484] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9712.746815] LustreError: 379660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9718.917720] Lustre: server umount lustre-MDT0001 complete [ 9722.900639] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 9724.057195] Key type lgssc unregistered [ 9724.187323] LNet: 384500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9724.190395] LNetError: 384500:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9724.202290] LNet: Removed LNI 192.168.202.103@tcp [ 9724.518496] Key type .llcrypt unregistered [ 9724.519980] Key type ._llcrypt unregistered [ 9743.059990] Key type ._llcrypt registered [ 9743.061404] Key type .llcrypt registered [ 9743.100272] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 9743.457414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9743.492352] alg: No test for adler32 (adler32-zlib) [ 9744.397354] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 9744.536897] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 9746.159145] Key type lgssc registered [ 9746.553179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9749.504870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9749.519449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9750.627187] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9750.633874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9750.695494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9752.275023] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9755.449625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9755.494633] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9755.511202] Lustre: 387274:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9755.609958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9757.042447] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9759.684737] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9761.633322] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9765.855903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9765.859257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9765.865457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9766.368505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9766.368794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9768.473410] Lustre: server umount lustre-MDT0000 complete [ 9769.918297] LustreError: 387264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144544 with bad export cookie 5462377567812170422 [ 9769.921495] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9769.923157] LustreError: 387264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9770.054352] Lustre: server umount lustre-MDT0001 complete [ 9773.646784] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 01:55:47 (1783144547) [ 9776.067862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9776.257420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9777.895451] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9781.012893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9781.153749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9782.639472] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9785.245245] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9787.219943] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9789.578376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9789.648646] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9789.722912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9791.769772] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9794.367439] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9799.345759] Lustre: Permanently deactivating lustre-OST0000 [ 9799.353063] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9808.884770] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9809.276176] LustreError: 392912:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 9809.279872] LustreError: 392912:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 9809.282635] LustreError: 392912:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9809.941692] LustreError: 392960:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 9809.944438] LustreError: 392960:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9816.787952] Lustre: server umount lustre-OST0000 complete [ 9817.752100] LustreError: 393209:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 9817.755119] LustreError: 393209:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9822.177955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9822.177974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9822.178856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9822.178860] Lustre: Skipped 1 previous similar message [ 9822.185931] Lustre: Skipped 1 previous similar message [ 9827.296135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9827.298889] Lustre: Skipped 2 previous similar messages [ 9828.348978] Lustre: server umount lustre-MDT0000 complete [ 9829.703248] LustreError: 389761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144603 with bad export cookie 5462377567812170989 [ 9829.706932] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9829.708641] LustreError: 389761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9839.130914] Lustre: server umount lustre-MDT0001 complete [ 9842.114075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9843.690408] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9844.658402] LustreError: 394365:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 9846.760435] Lustre: 394703:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.103@tcp [ 9847.430630] Lustre: server umount MGS complete [ 9851.050047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9851.203351] LustreError: 395425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9851.208872] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9851.210591] LustreError: 395425:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9851.212417] Lustre: Skipped 1 previous similar message [ 9851.231484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9852.537558] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9855.351862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9855.443892] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9856.868751] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9859.297571] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9861.104962] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9861.727944] Lustre: Permanently reactivating lustre-OST0000 [ 9861.733529] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9861.735987] Lustre: Skipped 2 previous similar messages [ 9888.735137] LustreError: 397255:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 9891.364042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9891.476480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9891.479344] Lustre: Skipped 1 previous similar message [ 9893.396329] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9895.733579] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9909.217529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9909.218037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9909.219797] LustreError: Skipped 1 previous similar message [ 9909.226273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9909.229700] Lustre: Skipped 1 previous similar message [ 9913.483524] Lustre: server umount lustre-OST0000 complete [ 9916.895770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9916.898721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9916.902541] Lustre: Skipped 1 previous similar message [ 9919.456053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9919.458674] Lustre: Skipped 2 previous similar messages [ 9920.788565] Lustre: server umount lustre-MDT0000 complete [ 9922.012172] LustreError: 395406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144696 with bad export cookie 5462377567812172025 [ 9922.015558] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9922.015621] LustreError: 395406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9922.119815] Lustre: server umount lustre-MDT0001 complete [ 9925.535555] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 9926.538131] Key type lgssc unregistered [ 9926.672305] LNet: 400144:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9926.674785] LNetError: 400144:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9926.687311] LNet: Removed LNI 192.168.202.103@tcp [ 9926.988497] Key type .llcrypt unregistered [ 9926.990432] Key type ._llcrypt unregistered [ 9932.749239] Key type ._llcrypt registered [ 9932.750971] Key type .llcrypt registered [ 9932.786467] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 9938.903190] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 9939.302784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9939.315807] alg: No test for adler32 (adler32-zlib) [ 9940.167110] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [ 9940.248414] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 9941.823129] Key type lgssc registered [ 9942.183220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9954.756316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9954.761602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9955.871668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9955.884258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9955.922713] Lustre: lustre-MDT0000: new disk, initializing [ 9955.952694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9955.959282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9957.630556] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9962.324132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9962.369673] Lustre: 403272:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9962.395883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9962.398159] Lustre: Skipped 1 previous similar message [ 9962.437093] Lustre: lustre-MDT0001: new disk, initializing [ 9962.470906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9962.484401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9962.488485] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9963.817892] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9967.562271] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9969.512251] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9972.005938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9972.104903] Lustre: lustre-OST0000: new disk, initializing [ 9972.107115] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9972.110000] Lustre: 405244:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9972.139640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9974.144638] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [ 9974.521356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9974.525433] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9974.567518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9978.363172] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9980.065277] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9980.158074] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9981.768805] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9981.848259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9984.992128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9984.995223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9984.997964] LustreError: Skipped 1 previous similar message [ 9985.004758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9985.009012] Lustre: Skipped 1 previous similar message [ 9988.766163] Lustre: server umount lustre-OST0000 complete [ 9993.184607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9993.187895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9993.194127] Lustre: Skipped 1 previous similar message [ 9993.196583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9995.232124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9995.232534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9996.325396] Lustre: server umount lustre-MDT0000 complete [ 9997.712347] LustreError: 403266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144771 with bad export cookie 16683571388590652457 [ 9997.712979] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9997.717396] LustreError: 403266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9997.834461] Lustre: server umount lustre-MDT0001 complete [10000.924657] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10001.503906] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 01:59:35 (1783144775) [10004.196926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10004.441582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10006.041118] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10009.189662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10009.330413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10010.920372] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10013.708183] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10015.971451] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10018.643310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10018.788199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10021.193033] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10023.928158] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10024.607424] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10030.603049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10034.149621] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10034.150362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10034.155523] Lustre: Skipped 1 previous similar message [10034.157693] Lustre: Skipped 1 previous similar message [10034.161549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10038.036811] Lustre: server umount lustre-OST0000 complete [10040.287837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10040.290769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10040.296472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10040.299176] Lustre: Skipped 1 previous similar message [10045.593377] Lustre: server umount lustre-MDT0000 complete [10046.893046] LustreError: 410467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144820 with bad export cookie 16683571388590653332 [10046.893366] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10046.897512] LustreError: 410467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10047.020161] Lustre: server umount lustre-MDT0001 complete [10050.677432] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [10051.766174] Key type lgssc unregistered [10051.890438] LNet: 411420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10051.892800] LNetError: 411420:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10051.904290] LNet: Removed LNI 192.168.202.103@tcp [10052.225514] Key type .llcrypt unregistered [10052.227173] Key type ._llcrypt unregistered [10059.958922] Key type ._llcrypt registered [10059.960493] Key type .llcrypt registered [10060.000687] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [10060.108379] Key type .llcrypt unregistered [10060.109497] Key type ._llcrypt unregistered [10067.231643] Key type ._llcrypt registered [10067.233062] Key type .llcrypt registered [10067.275780] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [10067.606379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10067.613957] alg: No test for adler32 (adler32-zlib) [10068.498808] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [10068.601364] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [10070.183144] Key type lgssc registered [10070.561131] Lustre: Echo OBD driver; http://www.lustre.org/ [10073.237917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10073.246078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10074.396380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10075.736798] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10078.632731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10078.770019] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10080.983803] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10083.654547] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10083.807968] LustreError: 413620:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10083.814754] LustreError: 413620:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10084.832622] LustreError: 413619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10084.836442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10084.838548] LustreError: 413619:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10086.029483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10086.170583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10087.578421] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10095.072251] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10095.075137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10095.080204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10096.610672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10096.611169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10096.615270] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10100.762350] Lustre: server umount lustre-OST0000 complete [10104.801240] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10104.803869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10104.808661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10104.810976] Lustre: Skipped 1 previous similar message [10106.848081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10106.848616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10108.181435] Lustre: server umount lustre-MDT0000 complete [10109.394229] LustreError: 413644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144883 with bad export cookie 10475258171578045846 [10109.396584] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10109.398505] LustreError: 413644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10109.511850] Lustre: server umount lustre-MDT0001 complete [10113.074679] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [10114.111787] Key type lgssc unregistered [10114.240604] LNet: 416776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10114.243729] LNetError: 416776:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10114.252367] LNet: Removed LNI 192.168.202.103@tcp [10114.581964] Key type .llcrypt unregistered [10114.583831] Key type ._llcrypt unregistered [10123.750689] Key type ._llcrypt registered [10123.751859] Key type .llcrypt registered [10123.793439] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [10124.145543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10124.199192] alg: No test for adler32 (adler32-zlib) [10125.046706] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [10125.137071] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [10126.719160] Key type lgssc registered [10127.111594] Lustre: Echo OBD driver; http://www.lustre.org/ [10129.796124] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10129.803166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10145.247478] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10150.367355] LustreError: 418143:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 4: rc = -110 [10176.991101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10178.929462] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10181.277579] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10183.428254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10183.587273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10183.609081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10184.887245] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10187.570799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10187.699949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10189.010176] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10191.329366] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10193.137163] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10196.961671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10196.963072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10196.965791] Lustre: Skipped 1 previous similar message [10201.555093] Lustre: server umount lustre-OST0000 complete [10203.103822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10203.106265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10203.110916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10203.113384] Lustre: Skipped 1 previous similar message [10207.200055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10207.200610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10208.984167] Lustre: server umount lustre-MDT0000 complete [10210.209216] LustreError: 418151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144984 with bad export cookie 6164245349078167601 [10210.212223] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10210.213610] LustreError: 418151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10210.324739] Lustre: server umount lustre-MDT0001 complete [10213.932747] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [10214.984908] Key type lgssc unregistered [10215.108325] LNet: 421712:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10215.110969] LNetError: 421712:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10215.116259] LNet: Removed LNI 192.168.202.103@tcp [10215.421376] Key type .llcrypt unregistered [10215.422506] Key type ._llcrypt unregistered [10224.356946] Key type ._llcrypt registered [10224.357975] Key type .llcrypt registered [10224.395375] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [10224.812028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10224.819470] alg: No test for adler32 (adler32-zlib) [10225.667033] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [10225.756306] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [10227.335153] Key type lgssc registered [10227.699128] Lustre: Echo OBD driver; http://www.lustre.org/ [10230.250157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10230.256350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10231.382973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10232.579993] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10235.157289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10235.273181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10236.644971] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10239.064312] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10240.807194] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10242.856678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10242.958101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10244.665111] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10246.928160] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10256.297569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10260.959954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10260.963789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10260.968442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10261.472426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10261.474680] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10264.405945] Lustre: server umount lustre-MDT0000 complete [10265.171262] LustreError: 423086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.3@tcp arrived at 1783145039 with bad export cookie 12143131030335080162 [10265.177389] LustreError: 423086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10265.838107] LustreError: 423837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145039 with bad export cookie 12143131030335079511 [10265.841577] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10265.842237] LustreError: 423837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10266.591985] LustreError: 423104:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10266.592750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10266.597326] LustreError: 423104:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10266.600672] Lustre: Skipped 1 previous similar message [10266.601267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10266.606385] Lustre: Skipped 3 previous similar messages [10270.623791] LustreError: 423843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10270.624162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10270.629571] LustreError: 423843:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10271.957103] Lustre: server umount lustre-OST0000 complete [10274.149461] Lustre: server umount lustre-MDT0001 complete [10277.569933] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [10278.548441] Key type lgssc unregistered [10278.684338] LNet: 426807:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10278.686619] LNetError: 426807:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10278.695255] LNet: Removed LNI 192.168.202.103@tcp [10278.986908] Key type .llcrypt unregistered [10278.988212] Key type ._llcrypt unregistered [10287.593479] Key type ._llcrypt registered [10287.595083] Key type .llcrypt registered [10287.633358] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [10288.035693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10288.045499] alg: No test for adler32 (adler32-zlib) [10288.896511] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [10288.984508] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [10290.559157] Key type lgssc registered [10290.910515] Lustre: Echo OBD driver; http://www.lustre.org/ [10293.379338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10293.384560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10294.543790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10295.883065] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10298.601028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10298.716429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10300.054027] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10302.445347] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10304.195124] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10306.352432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10306.454539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10308.260721] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10310.495605] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10320.807673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10324.447765] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10324.451170] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10324.455310] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10325.950198] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.3@tcp (stopping) [10325.985665] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10327.758776] Lustre: server umount lustre-MDT0001 complete [10331.104170] LustreError: 428928:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10331.104702] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10331.104872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10331.105652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10331.105657] Lustre: Skipped 2 previous similar messages [10331.110124] LustreError: 428928:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10335.434243] Lustre: server umount lustre-OST0000 complete [10336.225319] LustreError: 428199:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10336.708384] Lustre: server umount lustre-MDT0000 complete [10340.926748] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [10341.929223] Key type lgssc unregistered [10342.056625] LNet: 431903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10342.060388] LNetError: 431903:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10342.069332] LNet: Removed LNI 192.168.202.103@tcp [10342.385178] Key type .llcrypt unregistered [10342.386387] Key type ._llcrypt unregistered [10350.285552] Key type ._llcrypt registered [10350.286437] Key type .llcrypt registered [10350.327337] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [10356.253243] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [10356.648597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10356.654729] alg: No test for adler32 (adler32-zlib) [10357.501192] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [10357.588993] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [10359.167152] Key type lgssc registered [10359.516678] Lustre: Echo OBD driver; http://www.lustre.org/ [10371.410347] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [10376.143291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10376.148163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10377.222390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10377.233103] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10377.268164] Lustre: lustre-MDT0000: new disk, initializing [10377.286365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10377.291369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10378.491296] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10383.847492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10383.950542] Lustre: lustre-OST0000: new disk, initializing [10383.952753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10383.954132] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [10383.954262] Lustre: Skipped 1 previous similar message [10383.959592] Lustre: 436910:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10383.981252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10385.752162] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10389.357919] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10389.489218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10389.493858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10389.502148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10394.651601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10394.691835] Lustre: 436908:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10394.716857] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10394.748440] Lustre: lustre-MDT0001: new disk, initializing [10394.781572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10394.792752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [10394.795620] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [10396.064598] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10399.406074] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [10399.465368] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [10400.829770] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [10400.903763] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [10404.285990] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.3@tcp (stopping) [10404.321543] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10404.322364] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10404.327157] Lustre: Skipped 1 previous similar message [10408.654670] Lustre: server umount lustre-MDT0001 complete [10414.561288] LustreError: 435924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10414.566622] LustreError: 435924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10419.681275] LustreError: 435895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10419.687748] LustreError: 435895:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10424.800573] LustreError: 435900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10424.800773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10424.810449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10424.813411] Lustre: Skipped 2 previous similar messages [10426.965257] Lustre: server umount lustre-OST0000 complete [10434.480850] Lustre: server umount lustre-MDT0000 complete [10438.851646] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [10439.855466] Key type lgssc unregistered [10439.989338] LNet: 440648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10439.992395] LNetError: 440648:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10440.000401] LNet: Removed LNI 192.168.202.103@tcp [10440.292598] Key type .llcrypt unregistered [10440.293593] Key type ._llcrypt unregistered [10452.159558] Key type ._llcrypt registered [10452.160543] Key type .llcrypt registered [10452.195219] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [10452.284216] Key type .llcrypt unregistered [10452.285197] Key type ._llcrypt unregistered [10458.688075] Key type ._llcrypt registered [10458.689394] Key type .llcrypt registered [10458.725526] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [10459.105491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10459.111480] alg: No test for adler32 (adler32-zlib) [10459.950605] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [10460.027950] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [10461.607152] Key type lgssc registered [10461.942489] Lustre: Echo OBD driver; http://www.lustre.org/ [10470.047746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10470.052677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10471.126479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10471.136057] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10471.169669] Lustre: lustre-MDT0000: new disk, initializing [10471.187894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10471.193185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10472.533269] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10476.614096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10476.653828] Lustre: 443995:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10476.678423] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10476.680500] Lustre: Skipped 1 previous similar message [10476.714561] Lustre: lustre-MDT0001: new disk, initializing [10476.742898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10476.753212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10476.755898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10478.033665] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10481.605799] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10483.383880] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10485.532289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10485.628901] Lustre: lustre-OST0000: new disk, initializing [10485.630708] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10485.633293] Lustre: 445962:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10485.664155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10487.482086] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10491.000134] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10495.988823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10495.991869] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10496.023042] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10501.089267] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10501.090545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10501.094595] Lustre: Skipped 2 previous similar messages [10501.098654] Lustre: Skipped 2 previous similar messages [10504.664850] Lustre: server umount lustre-MDT0000 complete [10505.816067] LustreError: 444031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145279 with bad export cookie 16858335974986350677 [10505.818906] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10505.821458] LustreError: 444031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10505.913408] Lustre: server umount lustre-MDT0001 complete [10517.266066] Lustre: server umount lustre-OST0000 complete [10520.462360] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 02:08:14 (1783145294) [10522.809156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10522.962991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10524.182108] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10526.714140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10527.998501] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10530.138216] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10531.787083] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10546.913602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10546.972726] Lustre: Found index 0 for lustre-OST0000, updating log [10547.039259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10547.042454] Lustre: Skipped 1 previous similar message [10548.075224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10548.948173] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [10551.215581] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10553.311823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10553.314330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10553.318364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10556.384449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10556.385236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10556.389236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10558.480322] Lustre: server umount lustre-OST0000 complete [10572.768595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10572.768626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10572.769386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10572.769390] Lustre: Skipped 1 previous similar message [10572.773735] Lustre: Skipped 1 previous similar message [10577.043786] Lustre: server umount lustre-MDT0000 complete [10577.888135] LustreError: 447987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10577.892733] LustreError: 447987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10578.239117] LustreError: 447970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145352 with bad export cookie 16858335974986352056 [10578.243023] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10578.243327] LustreError: 447970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10578.350166] Lustre: server umount lustre-MDT0001 complete [10581.057396] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 02:09:14 (1783145354) [10609.567210] LNet: 1 local NIs in recovery (showing 1): 192.168.202.103@tcp [11160.399398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11160.558460] LustreError: 454388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11160.582894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11161.888559] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11164.725309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11166.274475] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11168.796477] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11170.591819] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11185.957603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11186.081756] Lustre: Found index 0 for lustre-OST0000, updating log [11186.192986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11186.195612] Lustre: Skipped 1 previous similar message [11188.262427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [11188.513827] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11190.970153] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11193.311944] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11193.314022] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11193.317899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11193.320363] Lustre: Skipped 1 previous similar message [11197.714177] Lustre: server umount lustre-OST0000 complete [11199.003100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11199.004082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11199.008982] Lustre: Skipped 2 previous similar messages [11199.012550] Lustre: Skipped 2 previous similar messages [11200.032494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11200.034480] Lustre: Skipped 1 previous similar message [11202.080367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11202.082452] Lustre: Skipped 3 previous similar messages [11206.176391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11206.179030] Lustre: Skipped 7 previous similar messages [11213.279167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11213.341611] Lustre: server umount lustre-MDT0000 complete [11213.344910] LustreError: 456419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11213.349425] LustreError: 456419:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11214.534293] LustreError: 454369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145988 with bad export cookie 16858335974986353127 [11214.537251] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11214.538920] LustreError: 454369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11214.646339] Lustre: server umount lustre-MDT0001 complete [11223.967264] LNet: 1 local NIs in recovery (showing 1): 192.168.202.103@tcp [11231.566332] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [11234.468390] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [11245.500799] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 02:20:19 (1783146019) [11251.130543] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [11252.127071] Key type lgssc unregistered [11252.254443] LNet: 461034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11252.258434] LNetError: 461034:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11318.821320] LNet: Removed LNI 192.168.202.103@tcp [11319.162279] Key type .llcrypt unregistered [11319.163499] Key type ._llcrypt unregistered [11325.558875] Key type ._llcrypt registered [11325.560024] Key type .llcrypt registered [11325.594184] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [11325.916503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11325.959766] alg: No test for adler32 (adler32-zlib) [11326.814599] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [11326.897456] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [11328.471121] Key type lgssc registered [11328.834600] Lustre: Echo OBD driver; http://www.lustre.org/ [11330.223188] LNet: Added LNI 192.168.202.103@tcp100 [8/256/0/180] [11330.506912] LNet: Added LNI 192.168.202.103@tcp101 [8/256/0/180] [11330.798660] LNet: Added LNI 192.168.202.103@tcp102 [8/256/0/180] [11331.089422] LNet: Added LNI 192.168.202.103@tcp103 [8/256/0/180] [11331.398963] LNet: Added LNI 192.168.202.103@tcp104 [8/256/0/180] [11331.688933] LNet: Added LNI 192.168.202.103@tcp105 [8/256/0/180] [11331.962408] LNet: Added LNI 192.168.202.103@tcp106 [8/256/0/180] [11332.242489] LNet: Added LNI 192.168.202.103@tcp107 [8/256/0/180] [11332.520123] LNet: Added LNI 192.168.202.103@tcp108 [8/256/0/180] [11332.826293] LNet: Added LNI 192.168.202.103@tcp109 [8/256/0/180] [11333.116194] LNet: Added LNI 192.168.202.103@tcp110 [8/256/0/180] [11333.392027] LNet: Added LNI 192.168.202.103@tcp111 [8/256/0/180] [11333.678929] LNet: Added LNI 192.168.202.103@tcp112 [8/256/0/180] [11333.972192] LNet: Added LNI 192.168.202.103@tcp113 [8/256/0/180] [11334.265538] LNet: Added LNI 192.168.202.103@tcp114 [8/256/0/180] [11334.542300] LNet: Added LNI 192.168.202.103@tcp115 [8/256/0/180] [11334.820860] LNet: Added LNI 192.168.202.103@tcp116 [8/256/0/180] [11335.105273] LNet: Added LNI 192.168.202.103@tcp117 [8/256/0/180] [11335.374373] LNet: Added LNI 192.168.202.103@tcp118 [8/256/0/180] [11335.648061] LNet: Added LNI 192.168.202.103@tcp119 [8/256/0/180] [11335.937205] LNet: Added LNI 192.168.202.103@tcp120 [8/256/0/180] [11336.242071] LNet: Added LNI 192.168.202.103@tcp121 [8/256/0/180] [11336.538980] LNet: Added LNI 192.168.202.103@tcp122 [8/256/0/180] [11336.824757] LNet: Added LNI 192.168.202.103@tcp123 [8/256/0/180] [11337.105391] LNet: Added LNI 192.168.202.103@tcp124 [8/256/0/180] [11337.386880] LNet: Added LNI 192.168.202.103@tcp125 [8/256/0/180] [11337.669371] LNet: Added LNI 192.168.202.103@tcp126 [8/256/0/180] [11337.939107] LNet: Added LNI 192.168.202.103@tcp127 [8/256/0/180] [11338.236255] LNet: Added LNI 192.168.202.103@tcp128 [8/256/0/180] [11338.530891] LNet: Added LNI 192.168.202.103@tcp129 [8/256/0/180] [11338.811233] LNet: Added LNI 192.168.202.103@tcp130 [8/256/0/180] [11339.097992] LNet: Added LNI 192.168.202.103@tcp131 [8/256/0/180] [11339.391510] LNet: Added LNI 192.168.202.103@tcp132 [8/256/0/180] [11339.671084] LNet: Added LNI 192.168.202.103@tcp133 [8/256/0/180] [11339.961678] LNet: Added LNI 192.168.202.103@tcp134 [8/256/0/180] [11340.247031] LNet: Added LNI 192.168.202.103@tcp135 [8/256/0/180] [11342.421264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11342.426230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11343.458743] Lustre: 464032:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.103@tcp: can't add NID 192.168.202.103@tcp130: rc = -75 [11343.488949] LNetError: 461498:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.103 [11343.498867] LNetError: Protocol error connecting to 192.168.202.103@tcp121 at host 192.168.202.103:988: is it running a compatible version of Lustre? [11343.504390] LNetError: Skipped 4 previous similar messages [11343.540535] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11343.551652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11343.601358] Lustre: lustre-MDT0000: new disk, initializing [11343.624728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11343.632164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11344.287305] LNet: 6 local NIs in recovery (showing 6): 192.168.202.103@tcp125, 192.168.202.103@tcp126, 192.168.202.103@tcp127, 192.168.202.103@tcp128, 192.168.202.103@tcp121, 192.168.202.103@tcp129 [11344.289270] LNetError: 464055:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.103 [11344.296721] LNetError: 464055:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [11344.309367] LNetError: Protocol error connecting to 192.168.202.103@tcp103 at host 192.168.202.103:988: is it running a compatible version of Lustre? [11344.314344] LNetError: Skipped 5 previous similar messages [11345.000375] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11349.020701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11349.070854] Lustre: 464047:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11349.098893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11349.100881] Lustre: Skipped 1 previous similar message [11349.140859] Lustre: lustre-MDT0001: new disk, initializing [11349.153845] Lustre: 464855:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.103@tcp: can't add NID 192.168.202.103@tcp130: rc = -75 [11349.157386] Lustre: 464855:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11349.199433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11349.210556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11349.213629] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11350.537769] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11352.247169] Lustre: 465236:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.103@tcp: can't add NID 192.168.202.103@tcp130: rc = -75 [11352.250121] Lustre: 465236:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11354.314209] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11356.155317] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11358.529809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11358.654190] Lustre: lustre-OST0000: new disk, initializing [11358.655842] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11358.658934] Lustre: 466026:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11358.697567] Lustre: 466031:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.103@tcp: can't add NID 192.168.202.103@tcp130: rc = -75 [11358.701070] Lustre: 466031:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [11358.720930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11359.870458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11359.873627] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11359.919649] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11360.642536] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11364.323618] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11370.466377] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11370.469086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11370.471189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11370.473258] Lustre: Skipped 1 previous similar message [11372.635683] Lustre: server umount lustre-OST0000 complete [11375.008554] LNetError: 464064:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.103 [11375.011787] LNetError: 464064:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [11375.014125] LNetError: Protocol error connecting to 192.168.202.103@tcp127 at host 192.168.202.103:988: is it running a compatible version of Lustre? [11375.072169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11375.074346] LustreError: Skipped 1 previous similar message [11375.075548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11375.079439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11375.081200] Lustre: Skipped 1 previous similar message [11377.056643] LNetError: 461497:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.103 [11377.063262] LNetError: Protocol error connecting to 192.168.202.103@tcp125 at host 192.168.202.103:988: is it running a compatible version of Lustre? [11380.704625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11380.707632] Lustre: Skipped 2 previous similar messages [11382.176311] LNetError: 464061:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.103 [11382.180630] LNetError: 464061:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [11382.184088] LNetError: Protocol error connecting to 192.168.202.103@tcp126 at host 192.168.202.103:988: is it running a compatible version of Lustre? [11382.188433] LNetError: Skipped 1 previous similar message [11385.825115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11385.828930] Lustre: Skipped 1 previous similar message [11388.383101] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11388.461237] Lustre: server umount lustre-MDT0000 complete [11389.739542] LustreError: 464041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146163 with bad export cookie 12955924519952440656 [11389.745246] LustreError: 464041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11389.745444] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11389.891117] Lustre: server umount lustre-MDT0001 complete [11390.368406] LNetError: 464067:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.103 [11390.373864] LNetError: 464067:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [11390.377958] LNetError: Protocol error connecting to 192.168.202.103@tcp125 at host 192.168.202.103:988: is it running a compatible version of Lustre? [11390.387021] LNetError: Skipped 3 previous similar messages [11394.333835] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [11395.550219] Key type lgssc unregistered [11395.711524] LNet: 467936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11395.715272] LNetError: 467936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11395.719100] LNet: Removed LNI 192.168.202.103@tcp [11396.767128] LNet: Removed LNI 192.168.202.103@tcp100 [11396.768483] LNetError: 467936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11396.770625] LNetError: 467936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11396.772927] LNet: Removed LNI 192.168.202.103@tcp101 [11397.791211] LNet: Removed LNI 192.168.202.103@tcp102 [11397.792924] LNetError: 467936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11397.795121] LNetError: 467936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11397.797825] LNet: Removed LNI 192.168.202.103@tcp103 [11398.815255] LNet: Removed LNI 192.168.202.103@tcp104 [11399.839209] LNet: Removed LNI 192.168.202.103@tcp105 [11399.841556] LNetError: 467936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11399.844814] LNetError: 467936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [11399.847750] LNet: Removed LNI 192.168.202.103@tcp106 [11400.863191] LNet: Removed LNI 192.168.202.103@tcp107 [11401.887224] LNet: Removed LNI 192.168.202.103@tcp108 [11402.911244] LNet: Removed LNI 192.168.202.103@tcp109 [11403.935190] LNet: Removed LNI 192.168.202.103@tcp110 [11403.936819] LNetError: 467936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11403.939799] LNetError: 467936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [11403.943315] LNet: Removed LNI 192.168.202.103@tcp111 [11404.959118] LNet: Removed LNI 192.168.202.103@tcp112 [11405.983167] LNet: Removed LNI 192.168.202.103@tcp113 [11407.007216] LNet: Removed LNI 192.168.202.103@tcp114 [11408.031222] LNet: Removed LNI 192.168.202.103@tcp115 [11409.055136] LNet: Removed LNI 192.168.202.103@tcp116 [11410.079220] LNet: Removed LNI 192.168.202.103@tcp117 [11411.103165] LNet: Removed LNI 192.168.202.103@tcp118 [11412.127163] LNet: Removed LNI 192.168.202.103@tcp119 [11412.129042] LNetError: 467936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11412.131719] LNetError: 467936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [11412.134423] LNet: Removed LNI 192.168.202.103@tcp120 [11413.151141] LNet: Removed LNI 192.168.202.103@tcp121 [11414.175113] LNet: Removed LNI 192.168.202.103@tcp122 [11415.199103] LNet: Removed LNI 192.168.202.103@tcp123 [11416.223105] LNet: Removed LNI 192.168.202.103@tcp124 [11417.247077] LNet: Removed LNI 192.168.202.103@tcp125 [11418.271092] LNet: Removed LNI 192.168.202.103@tcp126 [11419.295114] LNet: Removed LNI 192.168.202.103@tcp127 [11420.319127] LNet: Removed LNI 192.168.202.103@tcp128 [11421.343181] LNet: Removed LNI 192.168.202.103@tcp129 [11421.345504] LNet: Removed LNI 192.168.202.103@tcp130 [11421.347440] LNet: Removed LNI 192.168.202.103@tcp131 [11421.349409] LNet: Removed LNI 192.168.202.103@tcp132 [11421.351470] LNet: Removed LNI 192.168.202.103@tcp133 [11421.353699] LNet: Removed LNI 192.168.202.103@tcp134 [11421.361743] LNet: Removed LNI 192.168.202.103@tcp135 [11421.693851] Key type .llcrypt unregistered [11421.695110] Key type ._llcrypt unregistered [11428.855066] Key type ._llcrypt registered [11428.856159] Key type .llcrypt registered [11428.889568] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [11434.835895] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [11435.133336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11435.142106] alg: No test for adler32 (adler32-zlib) [11435.990475] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [11436.077462] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [11437.655113] Key type lgssc registered [11438.019138] Lustre: Echo OBD driver; http://www.lustre.org/ [11449.112930] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 02:23:42 (1783146222) [11454.940121] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [11455.991479] Key type lgssc unregistered [11456.131318] LNet: 471995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11456.134794] LNetError: 471995:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11456.143296] LNet: Removed LNI 192.168.202.103@tcp [11456.441302] Key type .llcrypt unregistered [11456.442991] Key type ._llcrypt unregistered [11463.164959] Key type ._llcrypt registered [11463.165960] Key type .llcrypt registered [11463.204063] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [11463.570549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11463.594234] alg: No test for adler32 (adler32-zlib) [11464.453517] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [11464.546828] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [11466.127153] Key type lgssc registered [11466.700940] Lustre: Echo OBD driver; http://www.lustre.org/ [11469.930711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11469.935808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11471.019703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11471.028733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11471.065459] Lustre: lustre-MDT0000: new disk, initializing [11471.089359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11471.095162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11472.395208] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11476.493964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11476.534248] Lustre: 473183:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11476.556743] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11476.558474] Lustre: Skipped 1 previous similar message [11476.596536] Lustre: lustre-MDT0001: new disk, initializing [11476.630609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11476.642456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11476.645615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11477.971535] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11481.695585] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11483.427728] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11485.706556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11485.813222] Lustre: lustre-OST0000: new disk, initializing [11485.815069] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11485.817685] Lustre: 475154:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11485.844824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11487.590214] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11491.047733] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11492.284598] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing lnet_if_list [11493.923371] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [11493.934123] LNet: Added LNI 192.168.202.103@tcp10 [8/256/0/180] [11494.401489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11494.405276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11494.448863] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11495.785383] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [11495.797956] LNet: Added LNI 192.168.202.103@tcp11 [8/256/0/180] [11497.070767] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [11497.081831] LNet: Added LNI 192.168.202.103@tcp12 [8/256/0/180] [11498.474652] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [11498.488482] LNet: Added LNI 192.168.202.103@tcp13 [8/256/0/180] [11499.792612] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [11499.803157] LNet: Added LNI 192.168.202.103@tcp14 [8/256/0/180] [11501.349373] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [11501.363918] LNet: Added LNI 192.168.202.103@tcp15 [8/256/0/180] [11502.984245] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [11502.998826] LNet: Added LNI 192.168.202.103@tcp16 [8/256/0/180] [11504.543568] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [11504.555104] LNet: Added LNI 192.168.202.103@tcp17 [8/256/0/180] [11506.235159] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [11506.244892] LNet: Added LNI 192.168.202.103@tcp18 [8/256/0/180] [11507.774350] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [11507.784566] LNet: Added LNI 192.168.202.103@tcp19 [8/256/0/180] [11509.557981] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [11509.573929] LNet: Added LNI 192.168.202.103@tcp20 [8/256/0/180] [11511.231467] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [11511.241432] LNet: Added LNI 192.168.202.103@tcp21 [8/256/0/180] [11512.871829] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [11512.881835] LNet: Added LNI 192.168.202.103@tcp22 [8/256/0/180] [11514.554784] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [11514.564981] LNet: Added LNI 192.168.202.103@tcp23 [8/256/0/180] [11516.272324] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [11516.282286] LNet: Added LNI 192.168.202.103@tcp24 [8/256/0/180] [11518.012197] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [11518.022976] LNet: Added LNI 192.168.202.103@tcp25 [8/256/0/180] [11519.772583] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [11519.784050] LNet: Added LNI 192.168.202.103@tcp26 [8/256/0/180] [11521.525615] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [11521.535172] LNet: Added LNI 192.168.202.103@tcp27 [8/256/0/180] [11523.393278] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [11523.402682] LNet: Added LNI 192.168.202.103@tcp28 [8/256/0/180] [11525.248555] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [11525.258517] LNet: Added LNI 192.168.202.103@tcp29 [8/256/0/180] [11527.166476] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [11527.177280] LNet: Added LNI 192.168.202.103@tcp30 [8/256/0/180] [11529.184854] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [11529.197678] LNet: Added LNI 192.168.202.103@tcp31 [8/256/0/180] [11531.185882] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [11531.198814] LNet: Added LNI 192.168.202.103@tcp32 [8/256/0/180] [11533.254857] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [11533.266714] LNet: Added LNI 192.168.202.103@tcp33 [8/256/0/180] [11535.347234] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [11535.359265] LNet: Added LNI 192.168.202.103@tcp34 [8/256/0/180] [11537.608112] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [11537.619059] LNet: Added LNI 192.168.202.103@tcp35 [8/256/0/180] [11539.736536] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [11539.748592] LNet: Added LNI 192.168.202.103@tcp36 [8/256/0/180] [11541.911160] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [11541.924510] LNet: Added LNI 192.168.202.103@tcp37 [8/256/0/180] [11544.078773] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [11544.088400] LNet: Added LNI 192.168.202.103@tcp38 [8/256/0/180] [11546.487089] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [11546.498030] LNet: Added LNI 192.168.202.103@tcp39 [8/256/0/180] [11548.917288] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [11548.927313] LNet: Added LNI 192.168.202.103@tcp40 [8/256/0/180] [11551.435044] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [11551.446299] LNet: Added LNI 192.168.202.103@tcp41 [8/256/0/180] [11553.898679] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [11553.910928] LNet: Added LNI 192.168.202.103@tcp42 [8/256/0/180] [11556.383662] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [11556.397087] LNet: Added LNI 192.168.202.103@tcp43 [8/256/0/180] [11558.750530] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [11558.760197] LNet: Added LNI 192.168.202.103@tcp44 [8/256/0/180] [11561.136224] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [11561.147107] LNet: Added LNI 192.168.202.103@tcp45 [8/256/0/180] [11563.450640] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [11563.460138] LNet: Added LNI 192.168.202.103@tcp46 [8/256/0/180] [11565.903083] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [11565.912783] LNet: Added LNI 192.168.202.103@tcp47 [8/256/0/180] [11568.322071] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [11568.332240] LNet: Added LNI 192.168.202.103@tcp48 [8/256/0/180] [11570.828854] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [11570.838609] LNet: Added LNI 192.168.202.103@tcp49 [8/256/0/180] [11573.490538] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [11573.503086] LNet: Added LNI 192.168.202.103@tcp50 [8/256/0/180] [11581.409441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11581.410965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11581.413278] Lustre: Skipped 2 previous similar messages [11581.417782] Lustre: Skipped 2 previous similar messages [11586.528678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11586.530707] Lustre: Skipped 2 previous similar messages [11587.221491] Lustre: server umount lustre-MDT0000 complete [11588.518311] LustreError: 473979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146362 with bad export cookie 3973271195321340029 [11588.521821] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11588.523283] LustreError: 473979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11588.631682] Lustre: server umount lustre-MDT0001 complete [11595.802314] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [11598.448826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11598.815352] LustreError: 472456:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bd07f76f x1869764349736704/t0(0) o250->MGC192.168.202.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11598.915696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11600.164515] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11602.832000] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11602.924345] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11602.953990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11604.226283] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11606.383537] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11607.007091] Lustre: 472459:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783146365/real 1783146365] req@000000008d6a8aed x1869764349736320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783146381 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11607.956454] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11608.036456] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11609.055297] Lustre: 472459:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783146367/real 1783146367] req@000000008249d4d7 x1869764349736448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783146383 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11611.108670] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11612.127135] Lustre: 472457:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783146370/real 1783146370] req@00000000698a9af6 x1869764349736576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783146386 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11614.175090] Lustre: 472458:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783146372/real 1783146372] req@00000000ffe87168 x1869764349736832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783146388 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11616.226486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11616.230685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11616.232485] Lustre: Skipped 1 previous similar message [11620.758623] Lustre: server umount lustre-OST0000 complete [11623.392187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11623.394518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11623.397782] Lustre: Skipped 1 previous similar message [11623.399607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11628.115804] Lustre: server umount lustre-MDT0000 complete [11629.399382] LustreError: 473177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146403 with bad export cookie 3973271195321342031 [11629.402761] LustreError: 473177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11629.402956] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11629.519361] Lustre: server umount lustre-MDT0001 complete [11633.019309] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [11634.054981] Key type lgssc unregistered [11634.216056] LNet: 502110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11634.220066] LNetError: 502110:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11634.223152] LNet: Removed LNI 192.168.202.103@tcp [11634.224391] LNet: Removed LNI 192.168.202.103@tcp10 [11634.225594] LNet: Removed LNI 192.168.202.103@tcp11 [11634.226852] LNet: Removed LNI 192.168.202.103@tcp12 [11634.228052] LNet: Removed LNI 192.168.202.103@tcp13 [11634.229173] LNet: Removed LNI 192.168.202.103@tcp14 [11634.230192] LNet: Removed LNI 192.168.202.103@tcp15 [11634.231252] LNet: Removed LNI 192.168.202.103@tcp16 [11634.232352] LNet: Removed LNI 192.168.202.103@tcp17 [11634.233421] LNet: Removed LNI 192.168.202.103@tcp18 [11634.235122] LNet: Removed LNI 192.168.202.103@tcp19 [11634.236340] LNet: Removed LNI 192.168.202.103@tcp20 [11634.237792] LNet: Removed LNI 192.168.202.103@tcp21 [11634.239227] LNet: Removed LNI 192.168.202.103@tcp22 [11634.241079] LNet: Removed LNI 192.168.202.103@tcp23 [11634.242427] LNet: Removed LNI 192.168.202.103@tcp24 [11634.243874] LNet: Removed LNI 192.168.202.103@tcp25 [11634.245207] LNet: Removed LNI 192.168.202.103@tcp26 [11634.246367] LNet: Removed LNI 192.168.202.103@tcp27 [11634.248250] LNet: Removed LNI 192.168.202.103@tcp28 [11634.250075] LNet: Removed LNI 192.168.202.103@tcp29 [11634.251181] LNet: Removed LNI 192.168.202.103@tcp30 [11634.252458] LNet: Removed LNI 192.168.202.103@tcp31 [11634.254620] LNet: Removed LNI 192.168.202.103@tcp32 [11634.256959] LNet: Removed LNI 192.168.202.103@tcp33 [11634.258848] LNet: Removed LNI 192.168.202.103@tcp34 [11634.260148] LNet: Removed LNI 192.168.202.103@tcp35 [11634.261294] LNet: Removed LNI 192.168.202.103@tcp36 [11634.262851] LNet: Removed LNI 192.168.202.103@tcp37 [11634.264327] LNet: Removed LNI 192.168.202.103@tcp38 [11634.265471] LNet: Removed LNI 192.168.202.103@tcp39 [11634.267076] LNet: Removed LNI 192.168.202.103@tcp40 [11634.268454] LNet: Removed LNI 192.168.202.103@tcp41 [11634.269613] LNet: Removed LNI 192.168.202.103@tcp42 [11634.270751] LNet: Removed LNI 192.168.202.103@tcp43 [11634.272143] LNet: Removed LNI 192.168.202.103@tcp44 [11634.273298] LNet: Removed LNI 192.168.202.103@tcp45 [11634.274495] LNet: Removed LNI 192.168.202.103@tcp46 [11634.275657] LNet: Removed LNI 192.168.202.103@tcp47 [11634.276715] LNet: Removed LNI 192.168.202.103@tcp48 [11634.277791] LNet: Removed LNI 192.168.202.103@tcp49 [11634.284347] LNet: Removed LNI 192.168.202.103@tcp50 [11634.566815] Key type .llcrypt unregistered [11634.568042] Key type ._llcrypt unregistered [11640.881150] Key type ._llcrypt registered [11640.883175] Key type .llcrypt registered [11640.920685] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [11646.880271] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [11647.289812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11647.298435] alg: No test for adler32 (adler32-zlib) [11648.153066] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [11648.239971] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [11649.823165] Key type lgssc registered [11650.192552] Lustre: Echo OBD driver; http://www.lustre.org/ [11661.422376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11661.427906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11662.508979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11662.519283] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11662.553871] Lustre: lustre-MDT0000: new disk, initializing [11662.573898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11662.580191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11663.919190] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11667.902041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11667.949130] Lustre: 505287:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11667.977894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11667.980901] Lustre: Skipped 1 previous similar message [11668.018081] Lustre: lustre-MDT0001: new disk, initializing [11668.048247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11668.059029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11668.063100] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11669.398807] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11673.130340] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11674.889924] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11677.385273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11677.459778] Lustre: *** cfs_fail_loc=90f, val=0*** [11677.489668] Lustre: lustre-OST0000: new disk, initializing [11677.491482] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11677.493990] Lustre: 507302:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11677.525190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11679.504675] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11681.271826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11681.275411] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11681.313810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11683.255485] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11686.369996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11686.372409] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11686.374573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11686.376184] Lustre: Skipped 1 previous similar message [11687.763821] Lustre: server umount lustre-OST0000 complete [11691.489078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11691.489779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11691.495508] Lustre: Skipped 1 previous similar message [11691.497971] Lustre: Skipped 1 previous similar message [11696.608200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11696.611343] Lustre: Skipped 2 previous similar messages [11701.728203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11701.730936] Lustre: Skipped 1 previous similar message [11703.263076] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11703.332462] Lustre: server umount lustre-MDT0000 complete [11704.641541] LustreError: 505280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146478 with bad export cookie 14311891286202292095 [11704.645401] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11704.646804] LustreError: 505280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11704.767443] Lustre: server umount lustre-MDT0001 complete [11708.522569] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [11709.528461] Key type lgssc unregistered [11709.660326] LNet: 509207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11709.662806] LNetError: 509207:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11709.670316] LNet: Removed LNI 192.168.202.103@tcp [11709.978480] Key type .llcrypt unregistered [11709.979560] Key type ._llcrypt unregistered [11717.251452] Key type ._llcrypt registered [11717.252470] Key type .llcrypt registered [11717.289250] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [11717.707971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11717.715301] alg: No test for adler32 (adler32-zlib) [11718.569725] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [11718.661035] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [11720.247163] Key type lgssc registered [11720.624531] Lustre: Echo OBD driver; http://www.lustre.org/ [11723.215235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11723.222067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11724.384447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11725.818381] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11728.698036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11728.838936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11730.293547] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11732.619502] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11734.368383] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11736.700973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11736.837209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11738.856683] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11741.299441] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11755.319857] loop0: detected capacity change from 0 to 204800000 [11755.379123] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11755.383602] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11755.387248] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11757.249837] loop0: detected capacity change from 0 to 204800000 [11757.307088] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11757.311268] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11757.314730] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11758.916770] loop0: detected capacity change from 0 to 204800000 [11760.686498] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11760.736125] Lustre: fs15246-OST0000: new disk, initializing [11760.737547] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [11760.739671] Lustre: 514068:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11760.757842] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11762.538378] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11766.135322] loop1: detected capacity change from 0 to 204800000 [11767.867545] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11767.908176] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [11767.932254] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [11767.965203] Lustre: fs15246-MDT0000: new disk, initializing [11768.012276] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11768.020883] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11768.062037] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11769.119757] Lustre: 515188:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000ffe87168 x1869764616027136/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1783146559 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [11769.222538] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11774.176775] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11774.179995] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11774.185630] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11779.044368] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11779.050302] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [11782.095710] Lustre: server umount fs15246-MDT0000 complete [11784.164053] LustreError: 511180:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11784.169565] LustreError: 511180:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11785.141715] Lustre: server umount fs15246-OST0000 complete [11789.282487] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11789.283148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11789.286326] Lustre: Skipped 1 previous similar message [11794.400369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11794.403265] Lustre: Skipped 2 previous similar messages [11794.776713] Lustre: server umount lustre-OST0000 complete [11799.521936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11799.522962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11799.525384] Lustre: Skipped 1 previous similar message [11802.275537] Lustre: server umount lustre-MDT0000 complete [11803.547513] LustreError: 510434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146577 with bad export cookie 4285194761611322625 [11803.550101] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11803.550881] LustreError: 510434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11803.664581] Lustre: server umount lustre-MDT0001 complete [11807.156123] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [11808.172170] Key type lgssc unregistered [11808.311290] LNet: 518281:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11808.313873] LNetError: 518281:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11808.324362] LNet: Removed LNI 192.168.202.103@tcp [11808.647060] Key type .llcrypt unregistered [11808.648310] Key type ._llcrypt unregistered [11815.602854] Key type ._llcrypt registered [11815.603973] Key type .llcrypt registered [11815.640658] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [11816.048944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11816.057480] alg: No test for adler32 (adler32-zlib) [11816.923495] Lustre: Lustre: Build Version: 2.17.54_83_gdf67fd6 [11817.018241] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [11818.599177] Key type lgssc registered [11818.961644] Lustre: Echo OBD driver; http://www.lustre.org/ [11831.348780] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [11834.489836] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [11845.895250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11845.901373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11846.983467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11846.993934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11847.027613] Lustre: lustre-MDT0000: new disk, initializing [11847.049484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11847.055790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11848.374283] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11852.399390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11852.441730] Lustre: 522715:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11852.463668] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11852.465461] Lustre: Skipped 1 previous similar message [11852.500878] Lustre: lustre-MDT0001: new disk, initializing [11852.531204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11852.541935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11852.544916] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11853.810946] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11857.405653] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11859.174105] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11861.330293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11861.432121] Lustre: lustre-OST0000: new disk, initializing [11861.433688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11861.436548] Lustre: 524683:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11861.465827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11863.296429] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all [11866.624607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11866.627767] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11866.654407] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11866.821759] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11868.211188] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11868.285740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11869.695425] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11869.774567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11871.714971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11871.715115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11871.715686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11871.719536] Lustre: Skipped 1 previous similar message [11871.724285] LustreError: Skipped 1 previous similar message [11872.468527] Lustre: server umount lustre-OST0000 complete [11876.833083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11876.833768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11876.837028] Lustre: Skipped 1 previous similar message [11876.841121] Lustre: Skipped 2 previous similar messages [11879.770302] Lustre: server umount lustre-MDT0000 complete [11880.950274] LustreError: 522752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146655 with bad export cookie 16436902613510356069 [11880.953648] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11880.954077] LustreError: 522752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11881.071178] Lustre: server umount lustre-MDT0001 complete [11889.241384] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [11892.688989] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [11902.871250] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11719 sec ========= 02:31:16 (1783146676) [11903.377390] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:31:17 (1783146677) === [11904.498364] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:31:18 (1783146678) === [11917.375607] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [11918.423883] Key type lgssc unregistered [11918.543343] LNet: 531312:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11918.545706] LNetError: 531312:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11918.558289] LNet: Removed LNI 192.168.202.103@tcp [11918.874981] Key type .llcrypt unregistered [11918.876885] Key type ._llcrypt unregistered