[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 527838561 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003211] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010020] pid_max: default: 32768 minimum: 301 [ 0.011186] LSM: Security Framework initializing [ 0.012083] Yama: becoming mindful. [ 0.013057] SELinux: Initializing. [ 0.014104] *** VALIDATE selinux *** [ 0.023504] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028893] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030124] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032014] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034069] *** VALIDATE tmpfs *** [ 0.035523] *** VALIDATE proc *** [ 0.037327] *** VALIDATE cgroup *** [ 0.038013] *** VALIDATE cgroup2 *** [ 0.039310] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041083] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043035] Spectre V2 : User space: Vulnerable [ 0.044010] Speculative Store Bypass: Vulnerable [ 0.046871] debug: unmapping init [mem 0xffffffff90e59000-0xffffffff90e60fff] [ 0.048163] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049624] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050028] ... version: 2 [ 0.051013] ... bit width: 48 [ 0.051987] ... generic registers: 4 [ 0.052013] ... value mask: 0000ffffffffffff [ 0.053014] ... max period: 00007fffffffffff [ 0.054013] ... fixed-purpose events: 3 [ 0.055013] ... event mask: 000000070000000f [ 0.056239] rcu: Hierarchical SRCU implementation. [ 0.058512] smp: Bringing up secondary CPUs ... [ 0.059646] x86: Booting SMP configuration: [ 0.060032] .... node #0, CPUs: #1 #2 #3 [ 0.063426] smp: Brought up 1 node, 4 CPUs [ 0.065013] smpboot: Max logical packages: 1 [ 0.066013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147073] node 0 deferred pages initialised in 77ms [ 0.150141] devtmpfs: initialized [ 0.151296] x86/mm: Memory block size: 128MB [ 0.154000] gcov: version magic: 0x41383552 [ 0.158247] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159085] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160314] pinctrl core: initialized pinctrl subsystem [ 0.161202] [ 0.161808] ************************************************************* [ 0.162015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163013] ** ** [ 0.164018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165019] ** ** [ 0.166016] ** This means that this kernel is built to expose internal ** [ 0.167013] ** IOMMU data structures, which may compromise security on ** [ 0.168016] ** your system. ** [ 0.169012] ** ** [ 0.170014] ** If you see this message and you are not debugging the ** [ 0.171016] ** kernel, report this immediately to your vendor! ** [ 0.172015] ** ** [ 0.173014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174017] ************************************************************* [ 0.175728] NET: Registered protocol family 16 [ 0.176513] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.177067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.180033] cpuidle: using governor menu [ 0.182764] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.185445] PCI: Using configuration type 1 for base access [ 0.187126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.196141] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.197042] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.198170] cryptd: max_cpu_qlen set to 1000 [ 0.199298] ACPI: Added _OSI(Module Device) [ 0.200028] ACPI: Added _OSI(Processor Device) [ 0.201023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.202014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.206302] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.209758] ACPI: Interpreter enabled [ 0.210068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.211013] ACPI: Using IOAPIC for interrupt routing [ 0.212118] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.213418] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.223643] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.224043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.225022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.226085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.228608] acpiphp: Slot [2] registered [ 0.229142] acpiphp: Slot [5] registered [ 0.230138] acpiphp: Slot [6] registered [ 0.231131] acpiphp: Slot [7] registered [ 0.232143] acpiphp: Slot [8] registered [ 0.233122] acpiphp: Slot [9] registered [ 0.234164] acpiphp: Slot [10] registered [ 0.235150] acpiphp: Slot [3] registered [ 0.236129] acpiphp: Slot [4] registered [ 0.237108] acpiphp: Slot [11] registered [ 0.238116] acpiphp: Slot [12] registered [ 0.239140] acpiphp: Slot [13] registered [ 0.240143] acpiphp: Slot [14] registered [ 0.241132] acpiphp: Slot [15] registered [ 0.242135] acpiphp: Slot [16] registered [ 0.243107] acpiphp: Slot [17] registered [ 0.244111] acpiphp: Slot [18] registered [ 0.245116] acpiphp: Slot [19] registered [ 0.246108] acpiphp: Slot [20] registered [ 0.247139] acpiphp: Slot [21] registered [ 0.248135] acpiphp: Slot [22] registered [ 0.249120] acpiphp: Slot [23] registered [ 0.250144] acpiphp: Slot [24] registered [ 0.251157] acpiphp: Slot [25] registered [ 0.252120] acpiphp: Slot [26] registered [ 0.253154] acpiphp: Slot [27] registered [ 0.254130] acpiphp: Slot [28] registered [ 0.255126] acpiphp: Slot [29] registered [ 0.256122] acpiphp: Slot [30] registered [ 0.257126] acpiphp: Slot [31] registered [ 0.258080] PCI host bridge to bus 0000:00 [ 0.259036] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.261029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.263030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.266024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.269031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.272103] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.273215] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.276496] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.280439] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.290870] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.296141] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.299025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.302025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.306029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.308593] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.310826] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.314057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.317814] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.322019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.334019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.337986] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.343979] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.352023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.360018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.375013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.384100] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.396024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.408018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.428026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.440000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.449030] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.456031] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.472020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.486228] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.493020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.504018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.521020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.531213] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.536012] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.548018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.561019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.573420] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.587019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.597017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.615030] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.629516] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.632442] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.634386] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.637392] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.641267] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.647061] iommu: Default domain type: Passthrough [ 0.649482] SCSI subsystem initialized [ 0.651160] ACPI: bus type USB registered [ 0.653121] usbcore: registered new interface driver usbfs [ 0.655123] usbcore: registered new interface driver hub [ 0.657103] usbcore: registered new device driver usb [ 0.659189] pps_core: LinuxPPS API ver. 1 registered [ 0.661011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.664080] PTP clock support registered [ 0.666126] EDAC MC: Ver: 3.0.0 [ 0.668127] PCI: Using ACPI for IRQ routing [ 0.670847] NetLabel: Initializing [ 0.671007] NetLabel: domain hash size = 128 [ 0.672008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.674093] NetLabel: unlabeled traffic allowed by default [ 0.676136] vgaarb: loaded [ 0.678259] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.679014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.687322] clocksource: Switched to clocksource kvm-clock [ 0.776564] VFS: Disk quotas dquot_6.6.0 [ 0.778474] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.781377] *** VALIDATE ramfs *** [ 0.782848] *** VALIDATE hugetlbfs *** [ 0.785426] pnp: PnP ACPI init [ 0.788193] pnp: PnP ACPI: found 6 devices [ 0.807051] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.810857] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.813138] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.815574] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.818400] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.821302] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.824629] NET: Registered protocol family 2 [ 0.827264] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.832832] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.836774] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.842344] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.846042] TCP: Hash tables configured (established 65536 bind 65536) [ 0.849040] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.851469] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.854745] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.857599] NET: Registered protocol family 1 [ 0.860220] RPC: Registered named UNIX socket transport module. [ 0.862538] RPC: Registered udp transport module. [ 0.864248] RPC: Registered tcp transport module. [ 0.865906] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.868466] NET: Registered protocol family 44 [ 0.870127] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.872536] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.875045] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.877473] PCI: CLS 0 bytes, default 64 [ 0.879721] Unpacking initramfs... [ 2.230244] debug: unmapping init [mem 0xffffa0d97cc54000-0xffffa0d97ffbffff] [ 2.234022] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.235712] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.237585] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.703198] Initialise system trusted keyrings [ 2.704492] Key type blacklist registered [ 2.706077] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.714433] zbud: loaded [ 2.717115] *** VALIDATE nfs *** [ 2.718102] *** VALIDATE nfs4 *** [ 2.719383] pstore: using deflate compression [ 2.722115] Platform Keyring initialized [ 2.817206] NET: Registered protocol family 38 [ 2.818855] Key type asymmetric registered [ 2.820103] Asymmetric key parser 'x509' registered [ 2.821526] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.825681] io scheduler mq-deadline registered [ 2.826941] io scheduler kyber registered [ 2.828122] io scheduler bfq registered [ 2.829494] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.831708] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.833810] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.835770] ACPI: Power Button [PWRF] [ 2.840602] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.846511] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.865690] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.876470] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.898888] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.924983] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.952616] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.956561] Non-volatile memory driver v1.3 [ 2.958362] Linux agpgart interface v0.103 [ 2.988298] virtio_blk virtio1: [vda] 149488 512-byte logical blocks (76.5 MB/73.0 MiB) [ 2.990322] vda: detected capacity change from 0 to 76537856 [ 3.002277] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.004401] vdb: detected capacity change from 0 to 1073741824 [ 3.018533] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.020278] vdc: detected capacity change from 0 to 2621440000 [ 3.041955] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.043837] vdd: detected capacity change from 0 to 2621440000 [ 3.064995] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.068047] vde: detected capacity change from 0 to 4294967296 [ 3.085311] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.087520] vdf: detected capacity change from 0 to 4294967296 [ 3.093264] libphy: Fixed MDIO Bus: probed [ 3.098108] usbcore: registered new interface driver usbserial_generic [ 3.099808] usbserial: USB Serial support registered for generic [ 3.101865] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.105474] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.107045] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.109122] mousedev: PS/2 mouse device common for all mice [ 3.112285] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.112326] rtc_cmos 00:05: RTC can wake from S4 [ 3.120304] rtc_cmos 00:05: registered as rtc0 [ 3.120497] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.121639] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.127477] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.128484] intel_pstate: CPU model not supported [ 3.133065] hid: raw HID events driver (C) Jiri Kosina [ 3.134555] usbcore: registered new interface driver usbhid [ 3.136076] usbhid: USB HID core driver [ 3.137178] drop_monitor: Initializing network drop monitor service [ 3.138825] Initializing XFRM netlink socket [ 3.140295] NET: Registered protocol family 10 [ 3.142464] Segment Routing with IPv6 [ 3.143532] NET: Registered protocol family 17 [ 3.144929] mpls_gso: MPLS GSO support [ 3.150712] RAS: Correctable Errors collector initialized. [ 3.152330] AVX version of gcm_enc/dec engaged. [ 3.153500] AES CTR mode by8 optimization enabled [ 3.215923] sched_clock: Marking stable (3215903510, 0)->(4185102432, -969198922) [ 3.219289] registered taskstats version 1 [ 3.220620] Loading compiled-in X.509 certificates [ 3.222163] zswap: loaded using pool lzo/zbud [ 3.240172] Key type big_key registered [ 3.248925] Key type encrypted registered [ 3.250139] ima: No TPM chip found, activating TPM-bypass! [ 3.251662] ima: Allocated hash algorithm: sha1 [ 3.253027] ima: No architecture policies found [ 3.254244] evm: Initialising EVM extended attributes: [ 3.255728] evm: security.selinux [ 3.256627] evm: security.ima [ 3.258167] evm: security.capability [ 3.259076] evm: HMAC attrs: 0x1 [ 3.260955] rtc_cmos 00:05: setting system clock to 2026-08-22 06:41:55 UTC (1787380915) [ 3.266212] debug: unmapping init [mem 0xffffffff91e03000-0xffffffff91ffffff] [ 3.268469] debug: unmapping init [mem 0xffffffff90b82000-0xffffffff90e58fff] [ 3.275115] Write protecting the kernel read-only data: 28672k [ 3.278292] debug: unmapping init [mem 0xffffffff8f203000-0xffffffff8f3fffff] [ 3.280385] debug: unmapping init [mem 0xffffffff8fb14000-0xffffffff8fbfffff] [ 3.310047] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.315808] systemd[1]: Detected virtualization kvm. [ 3.317186] systemd[1]: Detected architecture x86-64. [ 3.318598] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.346503] systemd[1]: No hostname configured. [ 3.347695] systemd[1]: Set hostname to . [ 3.349173] random: systemd: uninitialized urandom read (16 bytes read) [ 3.350703] systemd[1]: Initializing machine ID from random generator. [ 3.461757] random: systemd: uninitialized urandom read (16 bytes read) [ 3.463452] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.467253] random: systemd: uninitialized urandom read (16 bytes read) [ 3.469541] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.473271] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.046810] device-mapper: uevent: version 1.0.3 [ 4.049141] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.743950] virtio_net virtio0 ens2: renamed from eth0 [ 4.762659] random: fast init done [ 4.776583] scsi host0: ata_piix [ 4.830750] scsi host1: ata_piix [ 4.832480] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.835175] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.289192] dracut-initqueue[590]: RTNETLINK answers: File exists [ 9.764051] random: crng init done [ 9.765428] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.170589] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.397407] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.681291] SELinux: Disabled at runtime. [ 11.739516] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.749087] systemd[1]: Detected virtualization kvm. [ 11.751071] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.296412] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.300366] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.308391] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.313300] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.316899] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.324714] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.328796] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ 12.497916] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.831036] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.100629] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.161403] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.332301] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.368690] EDAC sbridge: Ver: 1.1.2 [ 15.364220] Key type dns_resolver registered [ 15.714328] NFS: Registering the id_resolver key type [ 15.719182] Key type id_resolver registered [ 15.723161] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg220-server login: [ 58.812094] hrtimer: interrupt took 3083426 ns [ 61.621301] libcfs: loading out-of-tree module taints kernel. [ 61.768716] Key type ._llcrypt registered [ 61.775283] Key type .llcrypt registered [ 61.982316] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 87.089139] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 89.205213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 89.235697] alg: No test for adler32 (adler32-zlib) [ 90.910119] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 92.171028] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 94.048202] Key type lgssc registered [ 95.929158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 118.856075] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 167.277321] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 183.376992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 183.459032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.816611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 184.845278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 185.005923] Lustre: lustre-MDT0000: new disk, initializing [ 185.143826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 185.174894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 190.185561] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.800391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 203.972640] Lustre: 6518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 204.013828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 204.022533] Lustre: Skipped 1 previous similar message [ 204.176347] Lustre: lustre-MDT0001: new disk, initializing [ 204.313193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 204.353186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 204.367277] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 209.093934] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.114630] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 225.419111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 225.726939] Lustre: lustre-OST0000: new disk, initializing [ 225.730899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 225.737127] Lustre: 8454:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 225.812072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 230.434576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 230.439951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 230.522320] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 232.057590] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.507215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 248.670764] Lustre: lustre-OST0001: new disk, initializing [ 248.674868] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 248.680995] Lustre: 9528:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 248.770704] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 256.942754] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 258.616684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 258.627821] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 258.705746] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 270.139697] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 277.670549] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 285.986476] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing check_logdir /tmp/testlogs/ [ 293.178533] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing yml_node [ 299.761882] Lustre: DEBUG MARKER: Client: 2.17.54.115 [ 303.181951] Lustre: DEBUG MARKER: MDS: 2.17.54.115 [ 306.183458] Lustre: DEBUG MARKER: OSS: 2.17.54.115 [ 308.395141] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 02:46:59 EDT 2026 [ 327.999782] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 330.554249] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 350.692472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 350.695064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 350.732460] Lustre: Skipped 1 previous similar message [ 354.501409] Lustre: server umount lustre-MDT0000 complete [ 358.910911] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381271 with bad export cookie 8732531091733052128 [ 358.919957] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 358.926291] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 360.931306] LustreError: 6524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 360.932102] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.932585] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 360.932590] Lustre: Skipped 3 previous similar messages [ 360.951215] LustreError: 6524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 361.016864] Lustre: Skipped 3 previous similar messages [ 363.938737] LustreError: 6525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 363.941660] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 363.970468] Lustre: Skipped 1 previous similar message [ 369.063327] LustreError: 8734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 369.067811] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 369.088879] LustreError: 8734:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 369.115762] Lustre: Skipped 1 previous similar message [ 373.220274] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 373.787750] Lustre: server umount lustre-MDT0001 complete [ 379.051271] Lustre: server umount lustre-OST0000 complete [ 384.129234] Lustre: server umount lustre-OST0001 complete [ 392.317427] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 403.577979] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 450.476931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.748948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 450.807619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 450.891501] Lustre: lustre-MDT0000: new disk, initializing [ 451.002429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 451.019916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 457.081807] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 470.828428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 470.951972] Lustre: 15089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 471.031496] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 471.035042] Lustre: Skipped 1 previous similar message [ 471.116412] Lustre: lustre-MDT0001: new disk, initializing [ 471.237225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 471.267068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 471.279384] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 477.235318] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 488.348204] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 495.281841] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 503.798579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 504.182460] Lustre: lustre-OST0000: new disk, initializing [ 504.186866] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 504.192867] Lustre: 17056:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 504.279767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 505.509547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 505.527482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 505.703473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 511.643749] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 524.354891] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 531.177861] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 531.403580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 537.726747] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 538.009443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 541.668235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 541.670800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 541.687672] LustreError: Skipped 1 previous similar message [ 541.688448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 541.758612] Lustre: Skipped 1 previous similar message [ 547.028115] Lustre: server umount lustre-OST0000 complete [ 551.906462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 551.911870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 551.927410] Lustre: Skipped 1 previous similar message [ 551.954108] Lustre: Skipped 4 previous similar messages [ 557.751499] Lustre: server umount lustre-MDT0000 complete [ 562.150157] LustreError: 15094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.180355] LustreError: 15094:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 562.399311] LustreError: 15893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381474 with bad export cookie 8732531091733053661 [ 562.406962] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 562.418008] LustreError: 15893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 563.115790] Lustre: server umount lustre-MDT0001 complete [ 569.031580] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 02:51:19 (1787381479) [ 572.403918] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 574.800312] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 02:51:25 (1787381485) [ 583.226062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.798204] LustreError: 19467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.855885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 589.412164] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 594.404159] LustreError: 19468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.429683] LustreError: 19468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 599.318252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 604.677745] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 614.581740] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 620.870161] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 628.255755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 635.675824] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 643.671931] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 654.305450] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 654.313397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 654.314995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 654.315020] Lustre: Skipped 2 previous similar messages [ 658.176853] Lustre: server umount lustre-OST0000 complete [ 664.556362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 664.579462] Lustre: Skipped 2 previous similar messages [ 668.801487] Lustre: server umount lustre-MDT0000 complete [ 669.672821] LustreError: 19468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 669.709547] LustreError: 19468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 673.208023] LustreError: 19448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381585 with bad export cookie 8732531091733054543 [ 673.211531] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 673.217043] LustreError: 19448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 673.513940] Lustre: server umount lustre-MDT0001 complete [ 686.274500] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 690.038766] Key type lgssc unregistered [ 690.523978] LNet: 23001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.533627] LNetError: 23001:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 690.590846] LNet: Removed LNI 192.168.202.120@tcp [ 692.333163] Key type .llcrypt unregistered [ 692.336545] Key type ._llcrypt unregistered [ 719.012299] Key type ._llcrypt registered [ 719.014449] Key type .llcrypt registered [ 719.149867] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 720.264312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 720.366664] alg: No test for adler32 (adler32-zlib) [ 721.508787] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 721.962296] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 723.736183] Key type lgssc registered [ 725.334131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 732.403838] loop: module loaded [ 732.517228] loop0: detected capacity change from 0 to 204800000 [ 732.668797] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 732.690559] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 733.154553] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 739.554765] loop0: detected capacity change from 0 to 204800000 [ 739.856881] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 739.883696] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 739.901974] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 745.685319] loop0: detected capacity change from 0 to 204800000 [ 751.226357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 751.251462] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 752.532419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 752.578538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 752.693143] Lustre: lustre-MDT0000: new disk, initializing [ 752.831126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 752.869443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 756.614780] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 765.629135] loop1: detected capacity change from 0 to 204800000 [ 771.998170] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 772.391344] Lustre: lustre-OST07c6: new disk, initializing [ 772.394674] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 772.398156] Lustre: Skipped 1 previous similar message [ 772.402090] Lustre: 26195:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 772.468693] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 779.552634] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 780.383657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 780.402228] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 780.441513] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 785.111842] Lustre: 25181:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 785.119721] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 785.124197] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 785.129608] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 785.135166] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 785.145424] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 799.461947] Lustre: 25179:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 799.480126] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 799.485822] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 799.502784] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 799.515930] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 799.523202] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 805.098951] Lustre: 25179:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 805.125041] Lustre: 25179:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 805.139730] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 805.167399] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 805.197590] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 805.222206] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 805.237598] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 805.253694] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 805.258578] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 805.263247] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 805.269363] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 805.273812] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.472049] Lustre: 25181:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 808.486603] Lustre: 25181:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 808.490866] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 808.499601] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.504328] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 808.508983] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.519144] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 808.531034] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.547669] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 808.553143] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.558255] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 808.565633] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 816.771136] Lustre: 25181:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 816.798807] Lustre: 25181:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 816.811410] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 816.828863] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 816.836829] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 816.850571] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 816.856637] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 816.871533] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 816.885381] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 816.898380] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 816.909464] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 816.914799] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 828.214712] Lustre: 25179:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 828.237448] Lustre: 25179:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 828.249566] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 828.262207] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 828.271551] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 828.287010] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 828.293926] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 828.303889] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 828.312533] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 828.323265] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 828.332039] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 828.338906] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 877.538878] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 877.548589] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 877.570604] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 882.660719] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 887.779631] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 888.288671] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 888.476647] Lustre: server umount lustre-OST07c6 complete [ 898.539023] Lustre: server umount lustre-MDT0000 complete [ 919.275172] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 922.570523] Key type lgssc unregistered [ 922.933295] LNet: 28754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.944520] LNetError: 28754:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.964458] LNet: Removed LNI 192.168.202.120@tcp [ 924.175706] Key type .llcrypt unregistered [ 924.178882] Key type ._llcrypt unregistered [ 945.939701] Key type ._llcrypt registered [ 945.942382] Key type .llcrypt registered [ 946.112823] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 947.028459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 947.066876] alg: No test for adler32 (adler32-zlib) [ 948.430604] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 948.688604] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 950.528206] Key type lgssc registered [ 951.606287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 960.119654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 960.138046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 961.718282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 967.156861] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 971.239669] LustreError: 30024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 971.253850] LustreError: 30024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 975.312648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 975.777709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 980.372828] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 988.873708] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 994.540803] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1001.410381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1001.855462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1005.944297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1008.099658] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1015.641569] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1035.753213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1035.761430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1035.762424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1035.768917] Lustre: Skipped 1 previous similar message [ 1035.794924] Lustre: Skipped 1 previous similar message [ 1039.168987] Lustre: server umount lustre-OST0000 complete [ 1045.985353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1046.016373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1046.023836] Lustre: Skipped 1 previous similar message [ 1049.704368] Lustre: server umount lustre-MDT0000 complete [ 1051.112732] LustreError: 31265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1051.143913] LustreError: 31265:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1053.332781] LustreError: 30006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381965 with bad export cookie 951989327000738077 [ 1053.342614] LustreError: 30006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1053.345358] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1053.621401] Lustre: server umount lustre-MDT0001 complete [ 1064.546269] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1066.848393] Key type lgssc unregistered [ 1067.246477] LNet: 33760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.257689] LNetError: 33760:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1067.274512] LNet: Removed LNI 192.168.202.120@tcp [ 1068.245113] Key type .llcrypt unregistered [ 1068.251978] Key type ._llcrypt unregistered [ 1087.860455] Key type ._llcrypt registered [ 1087.864243] Key type .llcrypt registered [ 1087.963441] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1089.257522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1089.331086] alg: No test for adler32 (adler32-zlib) [ 1090.404332] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 1090.760909] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 1092.450189] Key type lgssc registered [ 1093.791734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1101.909984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1101.935169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.474335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.051902] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1113.064851] LustreError: 35030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1113.084204] LustreError: 35030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1116.852271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.219474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1121.518321] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1128.797276] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1134.883908] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1141.773092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1142.105893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1147.766678] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1148.339049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1156.358975] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1168.868394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1168.876578] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1168.892108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1173.988431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1173.991393] Lustre: Skipped 2 previous similar messages [ 1179.106957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1179.118168] Lustre: Skipped 1 previous similar message [ 1181.152159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1181.401333] Lustre: server umount lustre-OST0000 complete [ 1188.833427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1188.841872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1188.855550] Lustre: Skipped 1 previous similar message [ 1188.864834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1194.471449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1194.477531] Lustre: Skipped 3 previous similar messages [ 1200.099085] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1200.475080] Lustre: server umount lustre-MDT0000 complete [ 1204.718138] LustreError: 35030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1204.747181] LustreError: 35030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1204.802312] LustreError: 35010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382117 with bad export cookie 16910247559109430996 [ 1204.819175] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1205.140391] Lustre: server umount lustre-MDT0001 complete [ 1216.551675] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1219.749849] Key type lgssc unregistered [ 1220.180672] LNet: 38581:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1220.187386] LNetError: 38581:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1221.224644] LNet: Removed LNI 192.168.202.120@tcp [ 1222.184990] Key type .llcrypt unregistered [ 1222.189428] Key type ._llcrypt unregistered [ 1250.110697] Key type ._llcrypt registered [ 1250.113776] Key type .llcrypt registered [ 1250.254897] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1251.534863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1251.592159] alg: No test for adler32 (adler32-zlib) [ 1252.821344] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 1253.147102] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 1254.936412] Key type lgssc registered [ 1256.334784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1264.654382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1264.707962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1266.295405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1271.514661] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1275.874709] LustreError: 39998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1275.892131] LustreError: 39998:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1280.560450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1280.919538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1285.534884] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1294.747445] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1300.744832] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1308.272486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1308.550628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1311.620688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1315.671276] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1322.653587] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1335.588735] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.20@tcp (stopping) [ 1336.290951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.294396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1336.332834] Lustre: Skipped 2 previous similar messages [ 1336.349822] Lustre: Skipped 2 previous similar messages [ 1339.161695] Lustre: server umount lustre-MDT0000 complete [ 1340.674473] LustreError: 39994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1343.527113] LustreError: 39978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382255 with bad export cookie 17379615760545142734 [ 1343.540757] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1343.546445] LustreError: 39978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1345.786350] LustreError: 39995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1345.809383] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.20@tcp (stopping) [ 1345.827139] LustreError: 39995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1346.540531] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1348.577622] LustreError: 39994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1348.579179] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1348.610853] LustreError: 39994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1348.647744] Lustre: Skipped 1 previous similar message [ 1350.203377] Lustre: server umount lustre-MDT0001 complete [ 1366.012471] Lustre: server umount lustre-OST0000 complete [ 1385.520829] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1388.756364] Key type lgssc unregistered [ 1389.207157] LNet: 43858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1389.228039] LNetError: 43858:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1389.269327] LNet: Removed LNI 192.168.202.120@tcp [ 1390.330231] Key type .llcrypt unregistered [ 1390.332205] Key type ._llcrypt unregistered [ 1416.124887] Key type ._llcrypt registered [ 1416.128541] Key type .llcrypt registered [ 1416.222778] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1417.804963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1417.851293] alg: No test for adler32 (adler32-zlib) [ 1419.063773] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 1419.410983] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 1421.088943] Key type lgssc registered [ 1422.262931] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1429.847249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1429.864783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1431.330346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1435.978593] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1441.763000] LustreError: 45279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1441.776126] LustreError: 45279:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1445.845402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1446.317330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1451.342937] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1459.717511] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1466.185693] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1473.700629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1473.972038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1477.054882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1480.097944] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1487.611905] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1500.957201] Lustre: lustre-OST0000: Not available for connect from 192.168.202.20@tcp (stopping) [ 1501.678161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1501.684242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1501.685858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1501.701935] Lustre: Skipped 1 previous similar message [ 1501.728335] Lustre: Skipped 1 previous similar message [ 1504.433269] Lustre: server umount lustre-OST0000 complete [ 1523.170923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1523.176380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1523.187580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1527.267245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1527.268472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1528.515321] Lustre: server umount lustre-MDT0000 complete [ 1532.389948] LustreError: 45275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1532.406212] LustreError: 45275:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1532.504123] LustreError: 45258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382444 with bad export cookie 8166419179955823699 [ 1532.514785] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1532.524574] LustreError: 45258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1539.186329] Lustre: server umount lustre-MDT0001 complete [ 1549.939845] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1552.937935] Key type lgssc unregistered [ 1553.233622] LNet: 48991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1553.245770] LNetError: 48991:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1553.266233] LNet: Removed LNI 192.168.202.120@tcp [ 1554.269669] Key type .llcrypt unregistered [ 1554.272914] Key type ._llcrypt unregistered [ 1579.171546] Key type ._llcrypt registered [ 1579.173585] Key type .llcrypt registered [ 1579.317889] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1581.248904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1581.279754] alg: No test for adler32 (adler32-zlib) [ 1582.444079] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 1582.740725] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 1584.504275] Key type lgssc registered [ 1586.097575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1594.662812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1594.690533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1596.094630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1601.559942] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1606.629154] LustreError: 50412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1606.645953] LustreError: 50412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1611.532877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1611.857637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1616.974752] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1626.163449] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1632.740620] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1641.671129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1642.302092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1649.100407] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1659.893538] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1665.850860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1676.318964] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1680.213390] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1692.896532] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-22 3h10m03s [ 1694.874892] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1698.795435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1698.808950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.828216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1699.941617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1699.952836] Lustre: Skipped 4 previous similar messages [ 1701.666451] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.20@tcp (stopping) [ 1703.402600] Lustre: server umount lustre-MDT0000 complete [ 1706.469091] LustreError: 50413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1706.499540] LustreError: 50413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1707.622880] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1708.793395] LustreError: 50408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1712.614526] LustreError: 52711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1712.648000] LustreError: 52711:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1715.045692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1715.153789] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1715.323272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1715.368247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1716.360577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1719.869440] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1723.261932] Lustre: DEBUG MARKER: Wait for df (36609) ... [ 1725.360607] Lustre: DEBUG MARKER: done [ 1729.510553] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1729.520842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1729.547280] Lustre: Skipped 2 previous similar messages [ 1729.554577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1729.557457] Lustre: Skipped 1 previous similar message [ 1730.532528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1730.543236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1737.190870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1737.208320] Lustre: Skipped 4 previous similar messages [ 1743.329348] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1743.552810] Lustre: server umount lustre-OST0000 complete [ 1752.550124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1752.560531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1752.571465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1752.603395] Lustre: Skipped 3 previous similar messages [ 1762.272237] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1762.452189] Lustre: server umount lustre-MDT0000 complete [ 1765.864705] LustreError: 50413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1765.882814] LustreError: 50413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1766.770868] LustreError: 50393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382679 with bad export cookie 6200888744459880588 [ 1766.774889] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1766.779715] LustreError: 50393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1767.162843] Lustre: server umount lustre-MDT0001 complete [ 1780.518803] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1783.178920] Key type lgssc unregistered [ 1783.419038] LNet: 55363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.434381] LNetError: 55363:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.466743] LNet: Removed LNI 192.168.202.120@tcp [ 1784.357073] Key type .llcrypt unregistered [ 1784.358877] Key type ._llcrypt unregistered [ 1827.570540] Key type ._llcrypt registered [ 1827.575551] Key type .llcrypt registered [ 1827.744953] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1829.348393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1829.495458] alg: No test for adler32 (adler32-zlib) [ 1830.612161] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 1830.865252] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 1832.552186] Key type lgssc registered [ 1833.748217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1842.956360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1842.971248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1844.196564] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1844.212975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1844.444341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1849.862636] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1859.523553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1859.682500] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1859.753145] Lustre: 57579:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1860.259261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1865.220088] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1873.510385] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1879.765933] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1887.294102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1887.584508] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1887.896233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1895.710986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1895.721784] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1904.605832] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1912.604982] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-22 3h13m43s [ 1914.657166] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1923.478266] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1926.672594] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1926.850862] LustreError: 57587:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1946.928868] Lustre: lustre-MDT0000: Client 9ab6dbde-e564-4697-ae27-12bb775f4701 (at 192.168.202.20@tcp) reconnecting [ 1966.936155] LustreError: 57587:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1969.789919] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1972.777928] Lustre: DEBUG MARKER: done [ 1982.433340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1982.437223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1982.448301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1982.950858] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1982.953097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1982.962847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1986.313231] Lustre: server umount lustre-OST0000 complete [ 1993.197232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1993.205924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1993.259340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1993.274803] Lustre: Skipped 1 previous similar message [ 1999.059521] Lustre: server umount lustre-MDT0000 complete [ 2004.007610] LustreError: 57570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382916 with bad export cookie 12437991095692851854 [ 2004.020091] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2004.021801] LustreError: 57570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2004.276940] Lustre: server umount lustre-MDT0001 complete [ 2017.511704] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 2020.553544] Key type lgssc unregistered [ 2020.967434] LNet: 61911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2020.972567] LNetError: 61911:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2020.994410] LNet: Removed LNI 192.168.202.120@tcp [ 2022.185578] Key type .llcrypt unregistered [ 2022.192307] Key type ._llcrypt unregistered [ 2074.684209] Key type ._llcrypt registered [ 2074.686821] Key type .llcrypt registered [ 2074.747565] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 2075.842211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2076.111647] alg: No test for adler32 (adler32-zlib) [ 2077.704837] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 2078.254826] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 2079.912163] Key type lgssc registered [ 2081.891492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2086.911196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2086.934386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2088.193621] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2088.216641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2088.329874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2089.515327] Lustre: Failing over lustre-MDT0000 [ 2089.753359] Lustre: server umount lustre-MDT0000 complete [ 2096.780436] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 03:16:47 (1787383007) [ 2103.332718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2103.719825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2108.685729] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2117.932918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2118.033301] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2118.066687] Lustre: 64711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2118.408388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2123.106973] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2130.300364] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2135.838754] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2143.245416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2143.408853] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2143.570807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2148.954961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2149.098185] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2156.059089] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2161.414110] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2169.319809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2169.324963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2169.347035] Lustre: Skipped 1 previous similar message [ 2169.368960] Lustre: Skipped 2 previous similar messages [ 2174.444397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2174.459551] Lustre: Skipped 2 previous similar messages [ 2175.029606] Lustre: server umount lustre-MDT0000 complete [ 2178.538224] LustreError: 66568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383090 with bad export cookie 3412847594255091010 [ 2178.553073] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2178.756831] Lustre: server umount lustre-MDT0001 complete [ 2181.818359] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2186.697772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2194.913856] Lustre: 63635:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787383091/real 1787383091] req@000000009dfe38d1 x1874206906162688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787383107 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2194.970827] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2194.987251] Lustre: Skipped 1 previous similar message [ 2195.824402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2199.008194] Lustre: 63636:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787383095/real 1787383095] req@00000000ce590b71 x1874206906162816/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787383111 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2200.096645] Lustre: 63636:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787383096/real 1787383096] req@0000000017ba1eb1 x1874206906162944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787383112 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2203.176373] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x2f5cdece101f0e25 [ 2203.188540] Lustre: MGC192.168.202.120@tcp: Connection restored to 0@lo (at 0@lo) [ 2203.803462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2203.905818] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2204.120307] Lustre: 63635:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787383100/real 1787383100] req@00000000052b98cb x1874206906163200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787383116 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2208.661474] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2210.276843] Lustre: 63635:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787383106/real 1787383106] req@000000003bd39601 x1874206906163328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787383122 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2210.282984] LustreError: 68482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2210.340293] LustreError: 68482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2215.395135] LustreError: 68482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2215.409709] LustreError: 68482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2216.496335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2216.863402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2221.681820] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2222.070558] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2224.182747] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2228.949497] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2234.770125] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2242.531137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2242.544026] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2242.553729] LustreError: 66562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2242.585411] Lustre: server umount lustre-MDT0000 complete [ 2244.578303] LustreError: 69217:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2244.585672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2244.618452] Lustre: Skipped 1 previous similar message [ 2246.033259] LustreError: 64703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383158 with bad export cookie 3412847594255093285 [ 2246.044086] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2246.049479] LustreError: 64703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2246.319969] Lustre: server umount lustre-MDT0001 complete [ 2252.245357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2262.074144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2265.058581] Lustre: 63636:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787383161/real 1787383161] req@000000008bfab965 x1874206906189696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787383177 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2265.074418] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2271.268568] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x2f5cdece101f1388 [ 2271.277363] Lustre: MGC192.168.202.120@tcp: Connection restored to 0@lo (at 0@lo) [ 2271.570939] LustreError: 66563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2271.590189] LustreError: 66563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2271.645496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2271.693341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2276.804562] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2280.962248] LustreError: 66563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2280.990311] LustreError: 66563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2284.081718] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2286.203610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2291.360959] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2299.773400] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2306.010758] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2313.299922] Lustre: server umount lustre-MDT0000 complete [ 2317.290130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2317.298995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2317.339800] LustreError: 66562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2317.355526] LustreError: 66562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2317.671966] LustreError: 64703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383229 with bad export cookie 3412847594255094664 [ 2317.679740] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2317.684429] LustreError: 64703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2318.276850] Lustre: server umount lustre-MDT0001 complete [ 2326.447976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2330.851540] Lustre: DEBUG MARKER: files compared the same [ 2343.464880] Lustre: server umount lustre-OST0000 complete [ 2358.908464] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 2361.883628] Key type lgssc unregistered [ 2362.204634] LNet: 74444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2362.224454] LNetError: 74444:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2363.241308] LNet: Removed LNI 192.168.202.120@tcp [ 2364.112190] Key type .llcrypt unregistered [ 2364.116764] Key type ._llcrypt unregistered [ 2389.373102] Key type ._llcrypt registered [ 2389.375287] Key type .llcrypt registered [ 2389.487511] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 2390.560445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2390.684453] alg: No test for adler32 (adler32-zlib) [ 2391.797545] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 2392.107396] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 2393.809133] Key type lgssc registered [ 2395.372823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2403.926823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2403.950437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2405.476509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2410.686302] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug +malloc all [ 2415.598237] LustreError: 75862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.610360] LustreError: 75862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2420.356666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2420.664630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2425.037487] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug +malloc all [ 2432.905978] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2438.920103] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2447.000547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2447.250522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2450.365152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2453.688421] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug +malloc all [ 2461.581454] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2476.002922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2476.018684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.034856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2478.815251] Lustre: server umount lustre-OST0000 complete [ 2486.241975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2486.264331] Lustre: Skipped 1 previous similar message [ 2486.270382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2486.273045] Lustre: Skipped 1 previous similar message [ 2489.438689] Lustre: server umount lustre-MDT0000 complete [ 2491.368335] LustreError: 75861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2491.407131] LustreError: 75861:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2492.804452] LustreError: 78643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383405 with bad export cookie 10073862381589601289 [ 2492.811471] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2493.035514] Lustre: server umount lustre-MDT0001 complete [ 2505.807985] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 2509.640912] Key type lgssc unregistered [ 2510.145769] LNet: 79396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2510.160493] LNetError: 79396:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2510.184185] LNet: Removed LNI 192.168.202.120@tcp [ 2511.254733] Key type .llcrypt unregistered [ 2511.259299] Key type ._llcrypt unregistered [ 2537.988719] Key type ._llcrypt registered [ 2537.990592] Key type .llcrypt registered [ 2538.074405] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 2539.199632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2539.295726] alg: No test for adler32 (adler32-zlib) [ 2540.348878] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 2540.525167] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 2542.169165] Key type lgssc registered [ 2543.444485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2552.231081] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2552.268350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2567.843576] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2572.960404] LustreError: 80788:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.120@tcp: failed processing log, type 4: rc = -110 [ 2599.649083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2605.513808] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2613.179268] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2621.246286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2621.401245] Lustre: *** cfs_fail_loc=706, val=0*** [ 2621.412481] LustreError: 80122:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009826e7cf x1874207390566656/t0(0) o250->MGC192.168.202.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2621.997236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2622.132615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2626.468178] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2627.050969] LustreError: 80814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.066630] LustreError: 80814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2632.162871] LustreError: 80813:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2634.216348] LustreError: 81875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2634.260847] LustreError: 81875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2634.569077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2634.923445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2640.145754] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2648.477539] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2655.272643] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2660.846851] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2660.855595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2660.858913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2660.888171] Lustre: Skipped 1 previous similar message [ 2665.956395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2665.967573] Lustre: Skipped 2 previous similar messages [ 2671.083933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2672.096153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2672.299808] Lustre: server umount lustre-OST0000 complete [ 2681.313665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2681.318945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2681.321564] Lustre: Skipped 1 previous similar message [ 2681.324694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2681.326440] Lustre: Skipped 2 previous similar messages [ 2682.661719] Lustre: server umount lustre-MDT0000 complete [ 2686.253430] LustreError: 80796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383598 with bad export cookie 16603683770796030319 [ 2686.258153] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2686.268242] LustreError: 80796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2686.606567] Lustre: server umount lustre-MDT0001 complete [ 2697.402456] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 2700.824404] Key type lgssc unregistered [ 2701.169989] LNet: 84408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2701.177852] LNetError: 84408:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2702.247366] LNet: Removed LNI 192.168.202.120@tcp [ 2703.211579] Key type .llcrypt unregistered [ 2703.213307] Key type ._llcrypt unregistered [ 2732.568785] Key type ._llcrypt registered [ 2732.585623] Key type .llcrypt registered [ 2732.770644] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 2734.546512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2734.651994] alg: No test for adler32 (adler32-zlib) [ 2735.857192] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 2736.206813] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 2738.056176] Key type lgssc registered [ 2739.534737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2748.896276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2748.928529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2754.836448] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2763.411794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2763.801710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2768.877298] LustreError: 86512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2768.904433] LustreError: 86512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2768.954724] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2772.964544] LustreError: 86515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2772.993823] LustreError: 86515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2777.317678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2777.512841] LustreError: 86512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2777.526602] LustreError: 86512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2777.530336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2782.690900] LustreError: 86510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2782.709168] LustreError: 86510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2783.732802] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2787.809332] LustreError: 86516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2787.835528] LustreError: 86516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2793.690569] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2793.736981] Lustre: MGS: Client d42b1bd2-be37-47d9-971a-ec7a0914c170 (at 0@lo) reconnecting [ 2793.753856] LustreError: 86526:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2794.103715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2794.173899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2799.753726] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2814.433982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2814.447394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2814.462922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2814.965508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2814.976830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2819.097927] Lustre: server umount lustre-OST0000 complete [ 2825.185654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2825.189227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2825.190763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2829.766971] Lustre: server umount lustre-MDT0000 complete [ 2830.306907] LustreError: 86511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2830.330181] LustreError: 86511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2834.289309] Lustre: server umount lustre-MDT0001 complete [ 2838.319866] Lustre: server umount MGS complete [ 2850.327735] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 03:29:21 (1787383761) [ 2883.139302] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 2894.659482] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 2929.535375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2934.084839] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2943.023494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2943.262282] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2943.298747] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2943.362869] Lustre: lustre-MDT0001: new disk, initializing [ 2943.449211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2943.465344] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2943.472205] Lustre: 93680:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2943.482696] LustreError: 93680:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2947.728075] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2957.742548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2958.042911] Lustre: lustre-OST0000: new disk, initializing [ 2958.046524] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2958.052551] Lustre: 94447:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2958.150529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2965.044290] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2967.651088] Lustre: 94876:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000002a58def5 x1874207596125952/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787383895 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2975.298460] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2982.524211] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2982.545933] Lustre: MGS: Client 4470623b-43f7-4739-91e8-09a4daead915 (at 0@lo) reconnecting [ 2982.615602] Lustre: 92948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2982.638597] LustreError: 93666:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2982.654569] LustreError: 93666:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2982.703224] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2982.813848] Lustre: lustre-MDT0000: new disk, initializing [ 2983.000835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2983.026734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2983.044478] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2984.101638] Lustre: 95517:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000923bcefc x1874207596132736/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787383912 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2988.137400] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2989.230912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2989.245241] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2989.352584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3014.626692] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3014.633576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3014.636432] Lustre: Skipped 2 previous similar messages [ 3014.654963] Lustre: Skipped 1 previous similar message [ 3019.746968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3019.784990] Lustre: Skipped 2 previous similar messages [ 3020.118455] Lustre: server umount lustre-OST0000 complete [ 3029.986848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3029.989520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3030.002767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3030.011320] Lustre: Skipped 1 previous similar message [ 3030.031837] Lustre: Skipped 1 previous similar message [ 3040.224196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3040.397237] Lustre: server umount lustre-MDT0000 complete [ 3051.001206] Lustre: server umount lustre-MDT0001 complete [ 3054.055041] Lustre: server umount MGS complete [ 3066.302571] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 03:32:57 (1787383977) [ 3085.629790] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 3088.748978] Key type lgssc unregistered [ 3089.023193] LNet: 98019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3089.034088] LNetError: 98019:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3089.046830] LNet: Removed LNI 192.168.202.120@tcp [ 3089.896798] Key type .llcrypt unregistered [ 3089.899443] Key type ._llcrypt unregistered [ 3108.335666] Key type ._llcrypt registered [ 3108.337183] Key type .llcrypt registered [ 3108.471583] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 3110.056774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3110.124589] alg: No test for adler32 (adler32-zlib) [ 3111.324471] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 3111.537296] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 3113.201845] Key type lgssc registered [ 3114.487606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3122.966721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3122.979072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3124.713415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3129.493491] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3133.937388] LustreError: 99240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3133.961481] LustreError: 99240:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3138.620552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3139.052280] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3139.339664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3144.203380] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3152.289093] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3158.424067] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3165.883401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3166.361296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3169.463913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3173.678869] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3182.740649] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3194.342172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3194.360963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3194.370206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3194.378986] Lustre: Skipped 1 previous similar message [ 3194.390780] Lustre: Skipped 1 previous similar message [ 3198.536236] Lustre: server umount lustre-OST0000 complete [ 3204.579882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3204.581435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3204.600026] Lustre: Skipped 1 previous similar message [ 3204.618084] Lustre: Skipped 1 previous similar message [ 3209.089788] Lustre: server umount lustre-MDT0000 complete [ 3209.708409] LustreError: 99240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3209.736099] LustreError: 99240:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3213.141982] LustreError: 99221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384125 with bad export cookie 14640244772431527722 [ 3213.152074] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3213.163267] LustreError: 99221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3213.513409] Lustre: server umount lustre-MDT0001 complete [ 3226.400224] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 3228.892320] Key type lgssc unregistered [ 3229.165957] LNet: 102774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3229.172097] LNetError: 102774:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3229.188324] LNet: Removed LNI 192.168.202.120@tcp [ 3230.049603] Key type .llcrypt unregistered [ 3230.052193] Key type ._llcrypt unregistered [ 3247.280613] Key type ._llcrypt registered [ 3247.290039] Key type .llcrypt registered [ 3247.495722] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 3248.598322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3248.858664] alg: No test for adler32 (adler32-zlib) [ 3249.868568] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 3250.227207] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 3251.944617] Key type lgssc registered [ 3253.168582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3260.166371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3260.169676] LustreError: 103859:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3260.175780] Lustre: Skipped 1 previous similar message [ 3260.175812] LustreError: 103858:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3260.191071] LustreError: 103859:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3260.195910] LustreError: 103859:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3260.200704] LustreError: 103859:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3260.218654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3261.895474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3270.338675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3270.679305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3274.965991] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3280.217768] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing lsmod [ 3283.188347] LustreError: 105067:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3283.196764] LustreError: 105068:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3283.207476] LustreError: 105067:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4989 [ 3283.211351] LustreError: 105067:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3283.217819] LustreError: 105067:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3283.235041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3283.692604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3289.060753] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3289.065293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3289.082033] Lustre: Skipped 1 previous similar message [ 3289.111095] Lustre: Skipped 2 previous similar messages [ 3294.182579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3294.192236] Lustre: Skipped 2 previous similar messages [ 3295.201735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3300.321606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3300.330032] Lustre: Skipped 3 previous similar messages [ 3301.345069] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3301.608076] Lustre: server umount lustre-MDT0000 complete [ 3305.187352] LustreError: 103875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384217 with bad export cookie 15944056602336281913 [ 3305.191307] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3305.208223] LustreError: 103875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3305.440929] LustreError: 103892:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3305.441130] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.462255] LustreError: 103892:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3305.464208] LustreError: 104330:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005f3910c0 x1874208135075712/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3305.481845] Lustre: Skipped 1 previous similar message [ 3305.482992] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3305.873681] Lustre: server umount lustre-MDT0001 complete [ 3312.238697] Lustre: server umount lustre-OST0000 complete [ 3323.006947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3323.529799] LustreError: 106133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3323.641493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3329.003603] LustreError: 106134:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3329.026409] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3333.025285] LustreError: 106133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3338.410821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3338.894359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3344.104916] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3353.044591] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3359.386677] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3367.171887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3367.527958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3369.604078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3374.537477] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3382.923903] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3393.505448] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3393.510291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3393.520271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3397.451952] Lustre: server umount lustre-OST0000 complete [ 3403.746789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3403.762212] Lustre: Skipped 1 previous similar message [ 3408.552507] Lustre: server umount lustre-MDT0000 complete [ 3408.864928] LustreError: 106133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3408.874236] LustreError: 106133:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3412.986115] LustreError: 106114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384325 with bad export cookie 15944056602336282627 [ 3412.994459] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3413.010477] LustreError: 106114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3413.415316] Lustre: server umount lustre-MDT0001 complete [ 3426.270970] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 3429.032836] Key type lgssc unregistered [ 3429.406945] LNet: 109667:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3429.415367] LNetError: 109667:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3429.429742] LNet: Removed LNI 192.168.202.120@tcp [ 3430.140933] Key type .llcrypt unregistered [ 3430.146320] Key type ._llcrypt unregistered [ 3453.655098] Key type ._llcrypt registered [ 3453.657550] Key type .llcrypt registered [ 3453.747769] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 3455.131614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3455.160810] alg: No test for adler32 (adler32-zlib) [ 3456.259569] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 3456.564810] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 3458.283199] Key type lgssc registered [ 3459.416230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3468.012446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3468.026260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3469.538505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3474.296862] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3479.027904] LustreError: 111084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3479.040040] LustreError: 111084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3483.663367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3483.961168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3489.105508] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3496.143452] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3504.408477] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3512.825581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3513.247867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3514.302738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3519.645963] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3528.248847] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3535.131586] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3539.388597] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3545.057630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3545.062638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3545.070527] LustreError: Skipped 1 previous similar message [ 3545.071735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3545.113614] Lustre: Skipped 1 previous similar message [ 3549.501078] Lustre: server umount lustre-OST0000 complete [ 3555.303243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3555.306072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3555.323340] Lustre: Skipped 1 previous similar message [ 3555.332873] Lustre: Skipped 2 previous similar messages [ 3559.538324] Lustre: server umount lustre-MDT0000 complete [ 3560.421727] LustreError: 111079:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3560.440931] LustreError: 111079:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3563.482347] LustreError: 111064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384475 with bad export cookie 16026380044656631260 [ 3563.497753] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3563.513065] LustreError: 111064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3564.082285] Lustre: server umount lustre-MDT0001 complete [ 3578.870473] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 3582.853861] Key type lgssc unregistered [ 3583.397274] LNet: 114726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3583.427062] LNetError: 114726:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3583.482109] LNet: Removed LNI 192.168.202.120@tcp [ 3585.358664] Key type .llcrypt unregistered [ 3585.360055] Key type ._llcrypt unregistered [ 3607.224457] Key type ._llcrypt registered [ 3607.226071] Key type .llcrypt registered [ 3607.384405] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 3608.746510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3608.933795] alg: No test for adler32 (adler32-zlib) [ 3609.984666] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 3610.230657] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 3611.976531] Key type lgssc registered [ 3613.296956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3622.844629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3622.864490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3624.542872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3629.342185] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3634.666072] LustreError: 115949:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3634.677402] LustreError: 115949:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3638.634580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3639.038842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3644.169175] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3652.718040] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3659.299324] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3667.057467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3667.411052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3669.493056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3675.024746] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3684.014174] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3694.052245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3694.055979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3694.069513] Lustre: Skipped 1 previous similar message [ 3694.079867] Lustre: Skipped 1 previous similar message [ 3699.253202] LustreError: 117772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3699.285203] LustreError: 117772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3699.652901] Lustre: server umount lustre-OST0000 complete [ 3705.831693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3705.851722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3705.888272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3709.411621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3709.416774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3711.397228] Lustre: server umount lustre-MDT0000 complete [ 3714.536212] LustreError: 115950:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3714.562826] LustreError: 115950:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3715.989685] LustreError: 115929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384628 with bad export cookie 15686749269232979470 [ 3716.005614] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3716.007576] LustreError: 115929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3716.045257] LustreError: 115261:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 3716.451300] Lustre: server umount lustre-MDT0001 complete [ 3729.372733] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 3732.720294] Key type lgssc unregistered [ 3732.974812] LNet: 119485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3732.980609] LNetError: 119485:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3732.997680] LNet: Removed LNI 192.168.202.120@tcp [ 3733.767648] Key type .llcrypt unregistered [ 3733.770948] Key type ._llcrypt unregistered [ 3760.323837] Key type ._llcrypt registered [ 3760.325586] Key type .llcrypt registered [ 3760.448938] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 3762.132107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3762.211280] alg: No test for adler32 (adler32-zlib) [ 3763.448425] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 3763.760813] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 3765.504452] Key type lgssc registered [ 3767.251995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3775.916936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3775.940977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3777.425935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3783.116374] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3787.747833] LustreError: 120900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3787.761863] LustreError: 120900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3792.880914] LustreError: 120901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3794.265728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3794.708722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3800.338730] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3810.058528] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3817.899903] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3826.749572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3827.150253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3834.839230] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3843.683975] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3847.683222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3856.004466] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3860.488342] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3860.496703] Lustre: Skipped 2 previous similar messages [ 3863.659611] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3863.713299] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3865.924450] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3865.937278] Lustre: Skipped 2 previous similar messages [ 3873.929911] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3873.935649] Lustre: Skipped 1 previous similar message [ 3880.560328] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.20@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3880.572173] Lustre: Skipped 2 previous similar messages [ 3881.059680] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.20@tcp 192.168.0.[2,10]@tcp [ 3891.962761] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.20/24@tcp in log lustre-MDT0000 [ 3891.971897] Lustre: Skipped 2 previous similar messages [ 3901.028448] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.20/24@tcp [ 3901.040355] Lustre: Skipped 1 previous similar message [ 3908.070133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3908.073762] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3908.077824] LustreError: Skipped 1 previous similar message [ 3908.137948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3909.097132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3909.098838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3909.473493] Lustre: server umount lustre-OST0000 complete [ 3914.211322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3914.243094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3914.252475] Lustre: Skipped 2 previous similar messages [ 3919.331436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3919.338288] Lustre: Skipped 1 previous similar message [ 3924.450623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3928.036138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3928.427524] Lustre: server umount lustre-MDT0000 complete [ 3929.573372] LustreError: 123128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3929.593345] LustreError: 123128:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3932.270117] LustreError: 123542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384844 with bad export cookie 16607048142542269375 [ 3932.275760] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3932.285539] LustreError: 123542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3932.751742] Lustre: server umount lustre-MDT0001 complete [ 3945.739627] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 3949.977965] Key type lgssc unregistered [ 3950.338210] LNet: 125859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3950.350141] LNetError: 125859:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3950.372796] LNet: Removed LNI 192.168.202.120@tcp [ 3951.885543] Key type .llcrypt unregistered [ 3951.887710] Key type ._llcrypt unregistered [ 3989.766310] Key type ._llcrypt registered [ 3989.777242] Key type .llcrypt registered [ 3989.938798] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 3991.789953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3991.964164] alg: No test for adler32 (adler32-zlib) [ 3993.242914] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 3993.650370] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 3995.465288] Key type lgssc registered [ 3996.987404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4005.322693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4005.363627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4007.063222] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4007.126354] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.20/24@tcp [ 4007.150083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4012.441155] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4017.639675] LustreError: 127424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4017.648520] LustreError: 127424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4022.773714] LustreError: 127423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4022.919288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4023.329262] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4023.352483] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.20/24@tcp [ 4023.406790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4028.339493] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4036.786870] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4044.039449] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4051.542077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4051.862240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4052.968103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4059.019214] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4067.908666] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4078.564398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4078.569685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4078.570149] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4078.579219] Lustre: Skipped 1 previous similar message [ 4078.611610] Lustre: Skipped 1 previous similar message [ 4083.695047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4083.700782] Lustre: Skipped 1 previous similar message [ 4084.250845] Lustre: server umount lustre-OST0000 complete [ 4088.807574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4088.819341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4088.823315] Lustre: Skipped 1 previous similar message [ 4088.836309] Lustre: Skipped 1 previous similar message [ 4093.928126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4094.561360] Lustre: server umount lustre-MDT0000 complete [ 4098.846377] LustreError: 127405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385011 with bad export cookie 1554652168290477350 [ 4098.848932] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4098.861593] LustreError: 127405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4099.484540] Lustre: server umount lustre-MDT0001 complete [ 4110.944892] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 4114.041182] Key type lgssc unregistered [ 4114.372419] LNet: 131015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4114.378605] LNetError: 131015:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4115.433069] LNet: Removed LNI 192.168.202.120@tcp [ 4116.462243] Key type .llcrypt unregistered [ 4116.464312] Key type ._llcrypt unregistered [ 4143.773152] Key type ._llcrypt registered [ 4143.775987] Key type .llcrypt registered [ 4143.922429] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 4159.652447] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 4160.802739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4161.082880] alg: No test for adler32 (adler32-zlib) [ 4162.211363] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 4162.460882] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 4164.176751] Key type lgssc registered [ 4165.511276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4202.729862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4202.751612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4203.965666] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4203.994417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4204.072435] Lustre: lustre-MDT0000: new disk, initializing [ 4204.119105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4204.126838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4208.350508] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4219.436101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4219.561204] Lustre: 134421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4219.656427] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4219.662069] Lustre: Skipped 1 previous similar message [ 4219.777709] Lustre: lustre-MDT0001: new disk, initializing [ 4219.935076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4219.968709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4219.978862] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4224.626373] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4234.584671] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4240.752391] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4247.620724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4247.855758] Lustre: lustre-OST0000: new disk, initializing [ 4247.858191] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4247.863973] Lustre: 136389:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4247.939087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4254.445188] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4256.828814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4256.842750] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4256.951992] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4265.169473] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4270.987406] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4271.191407] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4276.750749] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4276.999044] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4282.862307] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4282.869872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4282.878545] LustreError: Skipped 1 previous similar message [ 4282.879784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4282.930771] Lustre: Skipped 1 previous similar message [ 4285.369502] Lustre: server umount lustre-OST0000 complete [ 4291.553614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4291.566323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4291.589026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4291.597815] Lustre: Skipped 1 previous similar message [ 4293.094116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4293.095725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4296.297646] Lustre: server umount lustre-MDT0000 complete [ 4300.842957] LustreError: 134412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385213 with bad export cookie 11459879214626904156 [ 4300.856536] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4301.197142] Lustre: server umount lustre-MDT0001 complete [ 4311.204788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4311.775654] LustreError: 138606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4311.786935] LustreError: 138606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4311.892583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4316.701093] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4317.154097] LustreError: 138607:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4322.277233] LustreError: 138606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4327.238589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4327.393673] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4327.401775] Lustre: Skipped 1 previous similar message [ 4327.635165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4333.210889] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4342.644038] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4349.848435] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4357.468460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4357.618647] LustreError: 140430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4357.959377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4365.779563] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4376.193881] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4383.888519] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4384.331335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4389.615819] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4389.843195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4400.099369] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4400.363441] Lustre: lustre-OST0001: new disk, initializing [ 4400.372771] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4400.388822] Lustre: 141982:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4400.501958] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4406.915163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4406.928293] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4407.045961] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4407.384747] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4416.735829] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4416.941811] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4422.348666] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4422.542655] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4428.305646] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ba604825800.ost_server_uuid 50 [ 4430.282723] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9ba604825800.ost_server_uuid in IDLE state after 0 sec [ 4437.987320] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4437.995828] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4437.997467] LustreError: Skipped 1 previous similar message [ 4438.000897] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4438.038725] Lustre: Skipped 1 previous similar message [ 4440.186914] Lustre: server umount lustre-OST0001 complete [ 4443.118529] LustreError: 140428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4443.139497] LustreError: 140428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4444.129384] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4444.140992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4448.230918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4448.235234] Lustre: Skipped 4 previous similar messages [ 4450.002762] Lustre: server umount lustre-OST0000 complete [ 4455.907497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4455.919822] LustreError: Skipped 1 previous similar message [ 4455.928338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4455.947476] Lustre: Skipped 1 previous similar message [ 4468.192769] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4468.432703] Lustre: server umount lustre-MDT0000 complete [ 4468.706208] LustreError: 138606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4468.722568] LustreError: 138606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4471.845844] LustreError: 139342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385384 with bad export cookie 11459879214626905038 [ 4471.849497] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4471.854838] LustreError: 139342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4472.198910] Lustre: server umount lustre-MDT0001 complete [ 4492.937262] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 4496.243181] Key type lgssc unregistered [ 4496.575186] LNet: 144985:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4496.580582] LNetError: 144985:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4496.600736] LNet: Removed LNI 192.168.202.120@tcp [ 4497.370089] Key type .llcrypt unregistered [ 4497.372864] Key type ._llcrypt unregistered [ 4538.806373] Key type ._llcrypt registered [ 4538.807689] Key type .llcrypt registered [ 4538.938152] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 4555.386708] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 4556.958906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4556.989837] alg: No test for adler32 (adler32-zlib) [ 4558.005172] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 4558.238854] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 4559.890128] Key type lgssc registered [ 4561.343874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4599.480097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4599.507748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4600.743411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4600.763118] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4600.828148] Lustre: lustre-MDT0000: new disk, initializing [ 4600.881591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4600.904536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4604.784612] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4615.222871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4615.373487] Lustre: 149139:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4615.428689] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4615.432716] Lustre: Skipped 1 previous similar message [ 4615.496817] Lustre: lustre-MDT0001: new disk, initializing [ 4615.599575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4615.670332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4615.685761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4619.631458] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4630.351465] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4636.874017] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4643.474424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4643.839894] Lustre: lustre-OST0000: new disk, initializing [ 4643.844555] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4643.852302] Lustre: 151110:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4643.938946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4649.610420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4649.627795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4649.789412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4651.468669] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4663.360897] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4668.976604] Lustre: Failing over lustre-OST0000 [ 4669.133858] Lustre: server umount lustre-OST0000 complete [ 4685.794789] Lustre: 147367:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385582/real 1787385582] req@000000000299189b x1874209506626688/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787385598 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4685.795320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4685.833425] Lustre: 147367:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4686.820087] Lustre: 147366:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385582/real 1787385582] req@00000000fe0ea817 x1874209506626304/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787385598 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4686.861179] Lustre: 147366:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4689.842914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4690.339445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4690.374812] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4690.976939] Lustre: 147368:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385587/real 1787385587] req@00000000979f226e x1874209506627712/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787385603 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4696.505781] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4696.863195] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4696.863887] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4697.266968] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4698.593666] Lustre: 147365:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385592/real 1787385592] req@000000005e67ab49 x1874209506628992/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787385608 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4698.626803] Lustre: 147365:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4701.305644] Lustre: Failing over lustre-MDT0000 [ 4701.526442] Lustre: server umount lustre-MDT0000 complete [ 4701.669367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4701.687183] Lustre: Skipped 1 previous similar message [ 4706.790389] LustreError: 149153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4706.806138] LustreError: 149153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4711.906972] LustreError: 149152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4711.933821] LustreError: 149152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4717.031262] LustreError: 149149:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4717.060473] LustreError: 149149:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4720.537646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4720.787452] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4721.091430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4721.149343] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4722.430580] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4725.869127] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4726.254175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4726.261504] Lustre: Skipped 1 previous similar message [ 4726.287967] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4726.353287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4736.488206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4736.489499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4736.507064] Lustre: Skipped 3 previous similar messages [ 4736.516345] Lustre: Skipped 1 previous similar message [ 4740.577957] Lustre: server umount lustre-OST0000 complete [ 4746.726402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4746.733044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4746.741546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4750.978984] Lustre: server umount lustre-MDT0000 complete [ 4751.841659] LustreError: 149152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4751.884580] LustreError: 149152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4754.984383] LustreError: 149133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385667 with bad export cookie 15047724282801328225 [ 4754.985813] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4754.991540] LustreError: 149133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4755.476864] Lustre: server umount lustre-MDT0001 complete [ 4768.694820] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 4771.840626] Key type lgssc unregistered [ 4772.104669] LNet: 154786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4772.115410] LNetError: 154786:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4773.163439] LNet: Removed LNI 192.168.202.120@tcp [ 4774.105167] Key type .llcrypt unregistered [ 4774.106798] Key type ._llcrypt unregistered [ 4816.715389] Key type ._llcrypt registered [ 4816.718045] Key type .llcrypt registered [ 4816.838115] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 4818.187172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4818.372171] alg: No test for adler32 (adler32-zlib) [ 4819.666667] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 4820.084965] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 4821.840156] Key type lgssc registered [ 4823.333662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4833.388666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4833.400285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4835.071184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4841.611061] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4845.539599] LustreError: 156308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4845.560598] LustreError: 156308:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4850.673084] LustreError: 156309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4855.792369] LustreError: 156308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4856.347310] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4859.875261] LustreError: 156309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4861.391687] LustreError: 156338:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4861.410322] LustreError: 156338:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4861.429561] LustreError: 156338:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 26, retries 0, failed: rc = -5 [ 4862.473340] Lustre: server umount lustre-MDT0000 complete [ 4883.957222] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 4898.558235] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 4948.430312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4948.811427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4948.852640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4949.003705] Lustre: lustre-MDT0000: new disk, initializing [ 4949.093861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4949.116831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4954.443296] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4966.478900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4966.649566] Lustre: 159655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4966.752882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4966.758765] Lustre: Skipped 1 previous similar message [ 4966.834946] Lustre: lustre-MDT0001: new disk, initializing [ 4966.922030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4966.952277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4966.968671] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4972.528826] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4983.178967] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4989.896769] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4997.980331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4998.430425] Lustre: lustre-OST0000: new disk, initializing [ 4998.435966] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4998.442967] Lustre: 161626:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4998.572544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5005.818858] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5008.494216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5008.510456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5008.635085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5016.526744] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5021.520801] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5023.487890] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 04:05:34 (1787385934) [ 5321.185740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5321.192036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5321.205875] Lustre: Skipped 1 previous similar message [ 5321.222208] Lustre: Skipped 2 previous similar messages [ 5326.305643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5326.322784] Lustre: Skipped 2 previous similar messages [ 5329.248967] Lustre: server umount lustre-MDT0000 complete [ 5331.425067] LustreError: 159663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5331.452121] LustreError: 159663:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5334.103410] LustreError: 159648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386246 with bad export cookie 2090664622703981917 [ 5334.114058] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5334.117668] LustreError: 159648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5336.546174] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5336.567089] Lustre: Skipped 1 previous similar message [ 5336.571559] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5339.553793] LustreError: 159664:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5339.559211] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5339.589825] LustreError: 159664:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5340.685804] Lustre: server umount lustre-MDT0001 complete [ 5351.325825] Lustre: server umount lustre-OST0000 complete [ 5363.210741] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 5377.280598] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 5423.531997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5423.850472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5423.890652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5424.002412] Lustre: lustre-MDT0000: new disk, initializing [ 5424.097053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5424.117288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5429.425794] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5441.547351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5441.694401] Lustre: 166305:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5441.752240] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5441.761140] Lustre: Skipped 1 previous similar message [ 5441.855392] Lustre: lustre-MDT0001: new disk, initializing [ 5441.958990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5442.001248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5442.018150] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5447.535315] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5459.324927] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5466.143948] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5474.704877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5475.104871] Lustre: lustre-OST0000: new disk, initializing [ 5475.108752] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5475.115512] Lustre: 168272:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5475.237467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5482.332879] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5484.145489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5484.160877] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5484.295887] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5494.357716] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5499.773770] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5500.100556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5506.270685] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5506.453435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5510.127815] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5510.134582] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5510.139716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5510.175083] Lustre: Skipped 1 previous similar message [ 5514.997227] Lustre: server umount lustre-OST0000 complete [ 5520.364632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5520.369659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5520.407068] Lustre: Skipped 1 previous similar message [ 5520.424443] Lustre: Skipped 2 previous similar messages [ 5525.477147] LustreError: 168895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5525.945971] Lustre: server umount lustre-MDT0000 complete [ 5529.736297] LustreError: 166298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386441 with bad export cookie 2090664622704140747 [ 5529.742334] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5529.746379] LustreError: 166298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5530.086878] Lustre: server umount lustre-MDT0001 complete [ 5538.679024] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:14:09 (1787386449) [ 5556.895716] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 5568.086931] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 5607.635538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5607.898586] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5607.923637] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5607.981701] Lustre: lustre-MDT0000: new disk, initializing [ 5608.046233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5608.062912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5612.084283] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5622.558127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5622.686967] Lustre: 173033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5623.021466] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5627.631674] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5637.598043] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5644.197786] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5654.030616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5654.190155] Lustre: lustre-OST0000: new disk, initializing [ 5654.193397] Lustre: Skipped 1 previous similar message [ 5654.196874] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5654.201454] Lustre: Skipped 2 previous similar messages [ 5654.205522] Lustre: 175009:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5660.202075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5660.207295] Lustre: Skipped 1 previous similar message [ 5660.211097] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5660.340195] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5661.123642] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5671.785211] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5680.612466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5680.621197] LustreError: Skipped 1 previous similar message [ 5680.629344] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5680.661665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5693.409034] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5693.717102] Lustre: server umount lustre-OST0000 complete [ 5700.065134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5700.070760] LustreError: Skipped 1 previous similar message [ 5700.072749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5700.080301] Lustre: Skipped 1 previous similar message [ 5712.352140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5712.556249] Lustre: server umount lustre-MDT0000 complete [ 5715.880208] LustreError: 173024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386628 with bad export cookie 2090664622704141629 [ 5715.886119] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5715.887449] LustreError: 173024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5716.083098] Lustre: server umount lustre-MDT0001 complete [ 5727.208464] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:17:17 (1787386637) [ 5749.223357] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 5759.318844] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 5798.526304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5798.831935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5798.863584] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5798.982596] Lustre: lustre-MDT0000: new disk, initializing [ 5799.064077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5799.067232] Lustre: Skipped 2 previous similar messages [ 5799.079252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5804.309426] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5815.679865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5815.783313] Lustre: 179337:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5816.154644] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5820.189452] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5829.790736] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5835.886894] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5844.947221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5845.384660] Lustre: 181313:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5852.274116] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5852.433073] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5853.165774] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5864.436894] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5878.243659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5878.245533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5878.264296] Lustre: Skipped 1 previous similar message [ 5878.277481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5878.294728] Lustre: Skipped 14 previous similar messages [ 5879.418565] Lustre: server umount lustre-OST0000 complete [ 5889.727410] Lustre: server umount lustre-MDT0000 complete [ 5893.608834] LustreError: 180957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5893.627242] LustreError: 180957:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5894.467775] LustreError: 179329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386806 with bad export cookie 2090664622704142714 [ 5894.482577] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5894.492911] LustreError: 179329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5895.084797] Lustre: server umount lustre-MDT0001 complete [ 5907.900261] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 5911.427852] Key type lgssc unregistered [ 5911.693549] LNet: 183170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5911.700291] LNetError: 183170:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5911.725110] LNet: Removed LNI 192.168.202.120@tcp [ 5912.754619] Key type .llcrypt unregistered [ 5912.758222] Key type ._llcrypt unregistered [ 5937.999967] Key type ._llcrypt registered [ 5938.002209] Key type .llcrypt registered [ 5938.130249] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 5939.476818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5939.561781] alg: No test for adler32 (adler32-zlib) [ 5940.592773] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 5940.795544] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 5942.440946] Key type lgssc registered [ 5943.508477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5952.475367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5952.508192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5954.030434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5958.789375] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5964.258346] LustreError: 184586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5964.280262] LustreError: 184586:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5967.677170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5968.217241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5973.180577] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5984.586867] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5993.183921] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6000.217237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6000.734888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6007.398751] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6015.503498] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6024.303759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6045.154750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6045.164076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6045.172101] Lustre: Skipped 1 previous similar message [ 6045.187866] Lustre: Skipped 1 previous similar message [ 6050.284671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6050.293301] Lustre: Skipped 1 previous similar message [ 6051.463618] Lustre: server umount lustre-OST0000 complete [ 6055.394370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6055.407607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6055.414798] Lustre: Skipped 1 previous similar message [ 6055.432053] Lustre: Skipped 1 previous similar message [ 6060.523238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6060.533698] Lustre: Skipped 1 previous similar message [ 6061.640530] Lustre: server umount lustre-MDT0000 complete [ 6065.638663] LustreError: 184583:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6065.655204] LustreError: 184583:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6065.714288] LustreError: 186433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386977 with bad export cookie 10439542935336513430 [ 6065.731168] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6066.049758] Lustre: server umount lustre-MDT0001 complete [ 6077.393532] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 6080.165570] Key type lgssc unregistered [ 6080.464587] LNet: 188146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6080.469723] LNetError: 188146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6080.483146] LNet: Removed LNI 192.168.202.120@tcp [ 6081.207186] Key type .llcrypt unregistered [ 6081.209376] Key type ._llcrypt unregistered [ 6105.977257] Key type ._llcrypt registered [ 6105.981542] Key type .llcrypt registered [ 6106.095551] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 6107.038634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6107.060324] alg: No test for adler32 (adler32-zlib) [ 6108.306487] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 6108.517413] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 6110.184306] Key type lgssc registered [ 6111.300279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6118.910178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6118.925486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6120.301310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6124.508929] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6130.665618] LustreError: 189566:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6130.696271] LustreError: 189566:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6133.482948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6133.815848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6138.178339] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6145.793499] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6152.315350] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6159.059719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6159.538308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6165.207606] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6165.684530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6171.929873] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6181.346661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6181.357357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6181.379299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6183.394883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6183.399080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6183.414553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6183.418765] LustreError: Skipped 1 previous similar message [ 6186.770962] Lustre: server umount lustre-OST0000 complete [ 6193.694192] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9ba6042d0800.ost_server_uuid 50 [ 6195.270250] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6210.531024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6210.538971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6210.556586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6210.559164] Lustre: Skipped 2 previous similar messages [ 6213.092874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6213.095548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6215.131959] Lustre: server umount lustre-MDT0000 complete [ 6218.217039] LustreError: 192058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6218.244739] LustreError: 192058:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6219.462596] LustreError: 192145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387131 with bad export cookie 18372119590859359058 [ 6219.472075] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6219.475851] LustreError: 192145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6219.485928] LustreError: 190297:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 6219.495235] LustreError: 190297:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 6219.816796] Lustre: server umount lustre-MDT0001 complete [ 6228.534269] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 04:25:39 (1787387139) [ 6235.731475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6236.333985] LustreError: 193480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6236.456382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6240.490807] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6241.766936] LustreError: 193481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6245.864046] LustreError: 193480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6250.995468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6251.481530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6256.529473] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6265.382900] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6272.819232] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6280.504347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6280.831054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6281.849769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6287.786679] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6296.040499] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6303.745812] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6303.962088] Lustre: lustre-OST0001: new disk, initializing [ 6303.965850] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6303.976044] Lustre: 196325:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6304.132696] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6310.629441] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6311.984639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6311.989540] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6312.101221] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6319.918474] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6326.047945] Lustre: lustre-OST0000: Not available for connect from 192.168.202.20@tcp (stopping) [ 6326.058187] Lustre: Skipped 1 previous similar message [ 6327.267390] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6327.275134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6329.451984] Lustre: server umount lustre-OST0000 complete [ 6331.129432] LustreError: 195303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6331.151027] LustreError: 195303:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6338.607948] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6338.833659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6344.655252] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6345.080551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6348.259067] LustreError: 195303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6348.322981] LustreError: 195303:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 6363.644925] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6363.661022] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6363.670418] LustreError: Skipped 1 previous similar message [ 6363.672520] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6363.692111] Lustre: Skipped 2 previous similar messages [ 6363.732639] Lustre: Skipped 3 previous similar messages [ 6367.494411] Lustre: server umount lustre-OST0001 complete [ 6377.340756] Lustre: server umount lustre-MDT0000 complete [ 6381.671665] LustreError: 196324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387293 with bad export cookie 18372119590859360129 [ 6381.682293] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6381.682930] LustreError: 196324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6382.162897] Lustre: server umount lustre-MDT0001 complete [ 6406.841695] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 04:28:37 (1787387317) [ 6413.006432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6413.222589] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6413.248345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6413.472977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6418.995533] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6430.394170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6430.509171] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6430.545755] Lustre: 199766:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6435.770930] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6444.689309] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6450.376985] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6457.955613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6458.175425] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6466.020955] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6466.722789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6474.510897] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6481.865571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6481.996674] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6482.276438] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6482.278585] Lustre: Skipped 2 previous similar messages [ 6488.910328] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6491.833193] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6496.585942] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6502.377844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6502.386259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6502.398813] Lustre: Skipped 3 previous similar messages [ 6502.412287] Lustre: Skipped 3 previous similar messages [ 6506.194631] Lustre: server umount lustre-OST0000 complete [ 6507.504977] LustreError: 201620:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6507.548679] LustreError: 201620:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 6522.855187] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6528.663213] Lustre: server umount lustre-OST0001 complete [ 6538.211396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6538.213494] Lustre: Skipped 7 previous similar messages [ 6539.149544] Lustre: server umount lustre-MDT0000 complete [ 6543.575263] LustreError: 199759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387455 with bad export cookie 18372119590859361536 [ 6543.592289] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6543.594822] LustreError: 199759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6544.130663] Lustre: server umount lustre-MDT0001 complete [ 6572.808647] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 04:31:23 (1787387483) [ 6590.560377] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 6606.587235] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 6648.758229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6649.109575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6649.200075] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6649.326097] Lustre: lustre-MDT0000: new disk, initializing [ 6649.451112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6649.493419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6654.518772] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6667.413430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6667.641329] Lustre: 207881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6667.724305] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6667.729174] Lustre: Skipped 1 previous similar message [ 6667.840090] Lustre: lustre-MDT0001: new disk, initializing [ 6668.063472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6668.083159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6674.029091] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6684.176632] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6690.414781] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6697.418381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6697.682392] Lustre: lustre-OST0000: new disk, initializing [ 6697.685415] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6697.689836] Lustre: 209851:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6703.288740] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6704.196262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6704.214328] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6704.356144] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6713.203205] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6718.364825] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6718.720249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6724.088348] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6724.277921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6730.209043] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6730.210754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6730.215704] LustreError: Skipped 2 previous similar messages [ 6730.222362] Lustre: Skipped 4 previous similar messages [ 6730.226520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6730.232064] Lustre: Skipped 1 previous similar message [ 6732.473633] Lustre: server umount lustre-OST0000 complete [ 6742.689114] Lustre: server umount lustre-MDT0000 complete [ 6745.573400] LustreError: 207892:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6745.601768] LustreError: 207892:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 6746.621952] LustreError: 207874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387658 with bad export cookie 18372119590859363321 [ 6746.621952] LustreError: 207873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387658 with bad export cookie 18372119590859363321 [ 6746.622672] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6746.640486] LustreError: 207873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6746.927376] Lustre: server umount lustre-MDT0001 complete [ 6757.896422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6764.272420] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6776.160075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6780.330801] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6787.706614] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6794.783589] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6805.313641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6812.070958] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6820.282755] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6826.176212] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6829.857502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6834.294953] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6834.541878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6836.705664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6836.712866] LustreError: Skipped 1 previous similar message [ 6836.721380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6836.742137] Lustre: Skipped 2 previous similar messages [ 6842.900565] Lustre: server umount lustre-OST0000 complete [ 6850.432874] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6855.279854] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6884.943433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6890.481744] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6890.509802] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6892.699568] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6901.490660] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6903.289650] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6904.876981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6904.891089] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6909.928836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6909.943938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6909.956136] Lustre: Skipped 7 previous similar messages [ 6914.265405] Lustre: server umount lustre-OST0000 complete [ 6925.099404] Lustre: server umount lustre-MDT0000 complete [ 6925.282797] LustreError: 212068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6925.302559] LustreError: 212068:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6928.571659] LustreError: 212049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387840 with bad export cookie 18372119590859364203 [ 6928.578645] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6928.582630] LustreError: 212049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6928.986300] Lustre: server umount lustre-MDT0001 complete [ 6938.779797] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 04:37:29 (1787387849) [ 6945.884291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6946.504075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6946.506562] Lustre: Skipped 6 previous similar messages [ 6951.473254] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6959.563328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6964.474638] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6971.950541] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6978.659552] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6986.798883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6993.580651] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7002.920928] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7009.474466] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7009.898380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7016.149056] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7016.401477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7023.327816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7023.644331] Lustre: lustre-OST0001: new disk, initializing [ 7023.646832] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7023.652401] Lustre: 221433:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7032.085633] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7032.375643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7032.384603] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7032.468224] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7042.264875] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7047.935481] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7048.170580] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7053.500199] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7053.779159] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7057.896365] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7057.924387] Lustre: Skipped 6 previous similar messages [ 7070.176562] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7070.306952] Lustre: server umount lustre-OST0001 complete [ 7078.243141] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7078.500212] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7084.702869] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7084.970585] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7116.278769] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7121.912612] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7121.926871] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7121.933402] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7121.940765] Lustre: Skipped 1 previous similar message [ 7122.961258] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7132.787516] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7135.680448] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7142.376620] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7142.380530] LustreError: Skipped 2 previous similar messages [ 7144.829967] Lustre: server umount lustre-OST0001 complete [ 7167.986112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7168.003087] Lustre: Skipped 14 previous similar messages [ 7170.931992] Lustre: server umount lustre-OST0000 complete [ 7181.445349] Lustre: server umount lustre-MDT0000 complete [ 7183.330325] LustreError: 218057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7183.346020] LustreError: 218057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [ 7185.364376] LustreError: 221432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388097 with bad export cookie 18372119590859365169 [ 7185.367528] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7185.375820] LustreError: 221432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7191.856926] Lustre: server umount lustre-MDT0001 complete [ 7221.508664] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 04:42:11 (1787388131) [ 7230.194694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7230.528958] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7230.556567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7235.670623] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7245.212485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7245.319270] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7245.361243] Lustre: 226680:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7251.058749] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7259.481466] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7265.748280] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7274.287983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7274.526293] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7282.851808] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7291.418729] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7302.756527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7302.807289] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7311.003560] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7320.012652] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7321.905348] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7328.285709] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7328.807577] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7333.171385] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7333.390649] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7340.380578] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ba606e26000.ost_server_uuid 50 [ 7343.115772] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9ba606e26000.ost_server_uuid in IDLE state after 0 sec [ 7344.577515] Lustre: Permanently deactivating lustre-OST0001 [ 7344.611967] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7349.225147] Lustre: Permanently reactivating lustre-OST0001 [ 7349.262074] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7349.270591] Lustre: Skipped 2 previous similar messages [ 7357.415971] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7357.441101] Lustre: Skipped 7 previous similar messages [ 7369.184669] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7369.432751] Lustre: server umount lustre-OST0001 complete [ 7380.689067] Lustre: server umount lustre-OST0000 complete [ 7399.394852] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7399.718085] Lustre: server umount lustre-MDT0000 complete [ 7403.971045] LustreError: 226675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388316 with bad export cookie 18372119590859366457 [ 7403.975945] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7403.980797] LustreError: 226675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7404.552286] Lustre: server umount lustre-MDT0001 complete [ 7433.162305] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 04:45:43 (1787388343) [ 7442.940550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7443.294669] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7443.342558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7443.352568] Lustre: Skipped 2 previous similar messages [ 7449.319907] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7460.970150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7461.185290] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7461.258978] Lustre: 233377:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7461.539378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7461.546408] Lustre: Skipped 9 previous similar messages [ 7467.204883] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7476.563634] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7483.655893] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7491.262465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7491.515061] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7491.547849] Lustre: Permanently deactivating lustre-OST0000 [ 7491.572425] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7499.371299] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7499.388274] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7509.626372] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7518.619701] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7518.718840] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7526.765693] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7528.753682] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7536.253311] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7538.607711] Lustre: Permanently reactivating lustre-OST0000 [ 7538.631408] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7538.635893] Lustre: Skipped 2 previous similar messages [ 7538.901988] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7538.925744] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 40 seconds. I think it's dead, and I am evicting it. exp 00000000b75131b5, cur 1787388451 deadline 1787388441 last 1787388411 [ 7539.019546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7559.661075] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7559.674595] LustreError: Skipped 4 previous similar messages [ 7562.975287] Lustre: server umount lustre-OST0001 complete [ 7573.828281] Lustre: server umount lustre-OST0000 complete [ 7585.386828] Lustre: server umount lustre-MDT0000 complete [ 7590.406556] LustreError: 236260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388502 with bad export cookie 18372119590859368137 [ 7590.421953] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7590.426760] LustreError: 236260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7590.875887] Lustre: server umount lustre-MDT0001 complete [ 7604.864424] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 7608.877407] Key type lgssc unregistered [ 7609.247551] LNet: 238261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7609.253286] LNetError: 238261:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7609.267851] LNet: Removed LNI 192.168.202.120@tcp [ 7610.353875] Key type .llcrypt unregistered [ 7610.357183] Key type ._llcrypt unregistered [ 7640.078891] Key type ._llcrypt registered [ 7640.084798] Key type .llcrypt registered [ 7640.187867] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 7641.186775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7641.373947] alg: No test for adler32 (adler32-zlib) [ 7642.388727] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 7642.693639] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 7644.352854] Key type lgssc registered [ 7645.738990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7657.619708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7657.677619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7659.485676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7665.644889] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7669.737407] LustreError: 239731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7669.765970] LustreError: 239731:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7674.850772] LustreError: 239730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7676.629323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7676.724900] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7676.751181] Lustre: 239720:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7676.760783] Lustre: Permanently deactivating lustre-MDT0001 [ 7676.774704] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7677.105605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7682.359214] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7685.710234] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7692.288327] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7699.725730] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7707.920048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7708.335592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7715.075166] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7722.671030] LustreError: 241573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7722.723517] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7725.486341] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7728.101119] LustreError: 241574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7729.188505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7734.248749] LustreError: 241573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7734.267912] LustreError: 241573:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7735.444069] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7735.584295] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7740.964670] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7740.979748] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7743.687243] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7752.795169] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7757.082853] Lustre: Permanently reactivating lustre-MDT0001 [ 7757.105434] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7757.110740] Lustre: Skipped 2 previous similar messages [ 7759.552534] Lustre: lustre-MDT0001: Client f1b6feb4-14f5-40ba-9d6c-8c898b0e35e2 (at 192.168.202.20@tcp) reconnecting [ 7764.559830] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7764.580626] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7764.587421] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7764.623398] LustreError: 243442:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7764.626688] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7764.642322] LustreError: 243442:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7764.677699] LustreError: 243442:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.120@tcp: cfg command failed: rc = -108 [ 7764.687242] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7764.687242] [ 7764.708983] LustreError: 239722:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7766.768803] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7768.887576] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7776.188548] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7776.449321] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7783.229462] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7783.434570] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7786.528533] Lustre: Permanently deactivating lustre-MDT0001 [ 7786.561821] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7786.575370] Lustre: Skipped 2 previous similar messages [ 7793.436699] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7804.899840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7804.909782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7804.940317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7807.457523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7807.475758] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7807.485713] Lustre: Skipped 2 previous similar messages [ 7812.580933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7812.586075] Lustre: Skipped 6 previous similar messages [ 7817.184148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7817.491816] Lustre: server umount lustre-MDT0000 complete [ 7817.705560] LustreError: 239727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7817.746084] LustreError: 239727:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7822.600396] LustreError: 239709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388734 with bad export cookie 16520219253380446352 [ 7822.614733] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7822.621565] LustreError: 239709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7823.054286] Lustre: server umount lustre-MDT0001 complete [ 7828.365754] Lustre: server umount lustre-OST0000 complete [ 7844.362373] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 04:52:34 (1787388754) [ 7860.392138] Lustre: server umount lustre-OST0001 complete [ 7871.132242] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 7885.450596] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 7928.110780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7928.378388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7928.387703] Lustre: Skipped 2 previous similar messages [ 7928.418860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7928.513483] Lustre: lustre-MDT0000: new disk, initializing [ 7928.703815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7928.727634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7933.959482] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7946.592382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7946.734964] Lustre: 247997:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7946.776540] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7946.780265] Lustre: Skipped 1 previous similar message [ 7946.848173] Lustre: lustre-MDT0001: new disk, initializing [ 7947.000202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7947.075940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7947.080389] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7952.811717] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7963.256804] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7969.519217] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7977.875846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7978.328818] Lustre: lustre-OST0000: new disk, initializing [ 7978.332905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7978.340608] Lustre: 249968:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7978.466119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7986.595880] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7988.357045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7988.378178] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7988.539935] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7998.719270] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8002.390732] LustreError: 248004:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8012.428417] LustreError: 248004:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8012.465914] LustreError: 250738:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8013.398464] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8013.730631] Lustre: lustre-OST0001: new disk, initializing [ 8013.734845] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8013.746935] Lustre: 251083:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8013.919439] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8020.623442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8020.643658] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8020.780088] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8021.456192] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8022.568396] LustreError: 250738:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8032.903777] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8035.359920] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8041.441809] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8041.453039] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8041.459470] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8041.475873] Lustre: Skipped 1 previous similar message [ 8041.495228] Lustre: Skipped 1 previous similar message [ 8044.014991] Lustre: server umount lustre-OST0001 complete [ 8046.570358] LustreError: 249953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8046.593044] LustreError: 249953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8065.507480] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8065.521298] LustreError: Skipped 1 previous similar message [ 8065.528125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8065.559913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8065.582247] Lustre: Skipped 1 previous similar message [ 8069.819544] Lustre: server umount lustre-OST0000 complete [ 8075.234656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8075.254055] LustreError: Skipped 1 previous similar message [ 8075.266847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8075.292926] Lustre: Skipped 1 previous similar message [ 8075.300258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8075.302697] Lustre: Skipped 2 previous similar messages [ 8080.781494] Lustre: server umount lustre-MDT0000 complete [ 8082.420148] LustreError: 250738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8082.449590] LustreError: 250738:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 8085.491155] LustreError: 247990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388997 with bad export cookie 16520219253380449159 [ 8085.499553] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8092.129803] Lustre: server umount lustre-MDT0001 complete [ 8106.219303] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 8110.110212] Key type lgssc unregistered [ 8110.529184] LNet: 253126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8110.539953] LNetError: 253126:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8111.590400] LNet: Removed LNI 192.168.202.120@tcp [ 8112.667763] Key type .llcrypt unregistered [ 8112.671402] Key type ._llcrypt unregistered [ 8163.876097] Key type ._llcrypt registered [ 8163.879312] Key type .llcrypt registered [ 8164.119897] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 8166.053632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8166.165782] alg: No test for adler32 (adler32-zlib) [ 8167.404721] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 8167.758844] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 8169.521941] Key type lgssc registered [ 8171.024986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8181.488842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8181.511949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8182.983045] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8183.025464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8183.152548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8189.304462] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8199.323223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8199.546829] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8199.632577] Lustre: 255343:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8200.132770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8206.758127] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8216.419614] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8223.701972] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8232.582427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8232.958360] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8233.363386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8240.648358] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8241.373729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8250.181440] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8308.194109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8308.201072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8308.229529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8308.259103] Lustre: Skipped 1 previous similar message [ 8313.601817] Lustre: server umount lustre-OST0000 complete [ 8319.355466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8335.196183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8335.641910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8338.741962] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8338.765376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8338.775315] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8338.797430] Lustre: Skipped 1 previous similar message [ 8343.484862] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8352.763893] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8395.244353] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8395.251712] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8395.265037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8395.265571] Lustre: Skipped 2 previous similar messages [ 8400.373095] LustreError: 258939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8400.401663] LustreError: 258939:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8400.799806] Lustre: server umount lustre-OST0000 complete [ 8410.081549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8410.095013] LustreError: Skipped 1 previous similar message [ 8410.096852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8410.112047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8410.119926] Lustre: Skipped 1 previous similar message [ 8415.724403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8415.740237] Lustre: Skipped 2 previous similar messages [ 8420.320184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8420.836574] LustreError: 257875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8420.859879] LustreError: 257875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8422.687892] Lustre: server umount lustre-MDT0000 complete [ 8424.934173] LustreError: 256375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8424.950963] LustreError: 256375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8428.937065] LustreError: 257208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389341 with bad export cookie 311727432442912815 [ 8428.957744] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8428.961540] LustreError: 257208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8429.963992] Lustre: server umount lustre-MDT0001 complete [ 8445.270210] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 8450.108695] Key type lgssc unregistered [ 8450.735727] LNet: 260783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8450.745290] LNetError: 260783:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8450.770335] LNet: Removed LNI 192.168.202.120@tcp [ 8452.082255] Key type .llcrypt unregistered [ 8452.084571] Key type ._llcrypt unregistered [ 8481.212440] Key type ._llcrypt registered [ 8481.214218] Key type .llcrypt registered [ 8481.360467] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 8482.777279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8482.921911] alg: No test for adler32 (adler32-zlib) [ 8484.054529] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 8484.343791] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 8486.056332] Key type lgssc registered [ 8487.142875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8496.511390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8496.551969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8498.119741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8503.685756] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8508.391199] LustreError: 262204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8508.426435] LustreError: 262204:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8513.522127] LustreError: 262205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8514.743527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8515.305708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8520.277612] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8529.197599] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8535.652222] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8544.513190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8544.953281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8552.531916] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8562.021701] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8568.496521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8604.642529] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8604.649475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8604.666175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8604.672111] Lustre: Skipped 1 previous similar message [ 8604.699647] Lustre: Skipped 1 previous similar message [ 8608.262106] Lustre: server umount lustre-OST0000 complete [ 8614.881799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8614.885173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8614.897566] Lustre: Skipped 1 previous similar message [ 8620.007094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8620.018907] Lustre: Skipped 2 previous similar messages [ 8625.122857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8625.138851] Lustre: Skipped 1 previous similar message [ 8628.192379] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8628.935081] Lustre: server umount lustre-MDT0000 complete [ 8630.247631] LustreError: 262205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8630.283194] LustreError: 262205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8634.844450] LustreError: 262184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389547 with bad export cookie 13028266592824719915 [ 8634.855612] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8634.887782] LustreError: 262184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8635.728310] Lustre: server umount lustre-MDT0001 complete [ 8652.610119] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 8657.136553] Key type lgssc unregistered [ 8657.769252] LNet: 266538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8657.804231] LNetError: 266538:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8657.827624] LNet: Removed LNI 192.168.202.120@tcp [ 8659.573931] Key type .llcrypt unregistered [ 8659.594765] Key type ._llcrypt unregistered [ 8682.565661] Key type ._llcrypt registered [ 8682.568027] Key type .llcrypt registered [ 8682.712198] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 8683.821903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8683.902939] alg: No test for adler32 (adler32-zlib) [ 8685.421992] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 8685.656939] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 8687.384397] Key type lgssc registered [ 8689.164586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8698.368331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8698.395888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8700.020806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8706.475604] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8710.629108] LustreError: 267761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8710.685066] LustreError: 267761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8715.746702] LustreError: 267762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8718.796334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8719.209058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8725.827915] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8737.494815] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8746.032383] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8756.225885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8756.615893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8766.132049] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8769.919083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8776.300961] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8795.625246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8795.635287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8795.641072] LustreError: Skipped 1 previous similar message [ 8795.645841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8797.393138] Lustre: server umount lustre-OST0000 complete [ 8805.870438] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8805.878038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8805.884525] Lustre: Skipped 2 previous similar messages [ 8805.898158] Lustre: Skipped 2 previous similar messages [ 8808.919881] Lustre: server umount lustre-MDT0000 complete [ 8810.989241] LustreError: 268490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8811.019658] LustreError: 268490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8813.838683] LustreError: 267743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389726 with bad export cookie 4401101529722831422 [ 8813.845341] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8813.852568] LustreError: 267743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8814.394705] Lustre: server umount lustre-MDT0001 complete [ 8830.535557] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 8834.796307] Key type lgssc unregistered [ 8835.341968] LNet: 271445:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8835.360367] LNetError: 271445:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8835.394654] LNet: Removed LNI 192.168.202.120@tcp [ 8836.546790] Key type .llcrypt unregistered [ 8836.550924] Key type ._llcrypt unregistered [ 8872.584191] Key type ._llcrypt registered [ 8872.596918] Key type .llcrypt registered [ 8872.807379] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 8873.921057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8873.937063] alg: No test for adler32 (adler32-zlib) [ 8875.402603] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 8875.979592] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 8877.725210] Key type lgssc registered [ 8879.823429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8892.529398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8892.557408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8894.239602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8901.183374] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8904.681425] LustreError: 272867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8904.703249] LustreError: 272867:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8909.800788] LustreError: 272866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8914.334865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8914.604639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8920.882448] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8930.844605] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8938.465410] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8947.969727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8948.503792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8956.795985] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8963.986892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8967.009459] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8994.789700] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8994.801910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8994.819530] Lustre: Skipped 1 previous similar message [ 8994.838609] Lustre: Skipped 1 previous similar message [ 8999.911374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8999.920017] Lustre: Skipped 1 previous similar message [ 9000.676145] Lustre: server umount lustre-OST0000 complete [ 9005.029424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9005.034675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9005.037076] Lustre: Skipped 1 previous similar message [ 9005.042754] Lustre: Skipped 1 previous similar message [ 9010.146776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9010.153247] Lustre: Skipped 1 previous similar message [ 9015.272074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9015.285193] Lustre: Skipped 1 previous similar message [ 9019.360329] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9019.673433] Lustre: server umount lustre-MDT0000 complete [ 9020.406718] LustreError: 272867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9020.445323] LustreError: 272867:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9025.408391] LustreError: 274710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389937 with bad export cookie 3271298158876983773 [ 9025.414616] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9025.418037] LustreError: 274710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9026.041486] Lustre: server umount lustre-MDT0001 complete [ 9041.662675] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 9045.659922] Key type lgssc unregistered [ 9046.060390] LNet: 277175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9046.064886] LNetError: 277175:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9046.089051] LNet: Removed LNI 192.168.202.120@tcp [ 9047.420931] Key type .llcrypt unregistered [ 9047.423911] Key type ._llcrypt unregistered [ 9069.718615] Key type ._llcrypt registered [ 9069.723520] Key type .llcrypt registered [ 9069.867293] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 9071.676503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9071.810210] alg: No test for adler32 (adler32-zlib) [ 9073.067579] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 9073.433695] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 9075.312215] Key type lgssc registered [ 9077.363586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9085.043647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9085.054798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9086.406065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9092.633590] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9096.686633] LustreError: 278416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9096.738343] LustreError: 278416:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9101.795215] LustreError: 278418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9105.480876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9105.894574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9112.193508] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9123.090807] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9130.399281] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9138.915646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9139.446484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9147.898575] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9155.853863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9157.458784] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9172.453491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9172.463026] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9172.484498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9176.546100] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9176.558280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9176.563462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9176.603697] Lustre: Skipped 1 previous similar message [ 9181.669616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9181.681925] Lustre: Skipped 1 previous similar message [ 9186.273038] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9186.524299] Lustre: server umount lustre-OST0000 complete [ 9191.946561] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9192.005540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9192.010557] Lustre: Skipped 1 previous similar message [ 9197.025831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9197.042545] Lustre: Skipped 1 previous similar message [ 9197.543286] Lustre: server umount lustre-MDT0000 complete [ 9201.601153] LustreError: 278380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390113 with bad export cookie 8429529943586513849 [ 9201.608604] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9201.609405] LustreError: 278380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9201.872319] Lustre: server umount lustre-MDT0001 complete [ 9216.895003] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 9219.591399] Key type lgssc unregistered [ 9219.881839] LNet: 282122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9219.885728] LNetError: 282122:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9219.896876] LNet: Removed LNI 192.168.202.120@tcp [ 9220.571718] Key type .llcrypt unregistered [ 9220.573520] Key type ._llcrypt unregistered [ 9234.558571] Key type ._llcrypt registered [ 9234.560784] Key type .llcrypt registered [ 9234.676324] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9253.275894] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 9268.639467] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 9269.768563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9269.950414] alg: No test for adler32 (adler32-zlib) [ 9271.015863] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 9271.237761] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 9272.937469] Key type lgssc registered [ 9274.310525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9316.145221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9316.179663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9317.485370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9317.508336] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9317.580475] Lustre: lustre-MDT0000: new disk, initializing [ 9317.666710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9317.680308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9322.678871] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9335.588980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9335.738974] Lustre: 285677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9335.778428] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9335.781542] Lustre: Skipped 1 previous similar message [ 9335.832787] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9335.867410] Lustre: lustre-MDT0001: new disk, initializing [ 9335.951941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9335.986442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9336.000833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9340.479429] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9351.790784] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9359.067394] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9367.479181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9367.788983] Lustre: lustre-OST0000: new disk, initializing [ 9367.792560] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9367.798578] Lustre: 287648:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9367.908860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9374.841853] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9375.261352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9375.266220] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9375.390517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9385.573660] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9391.474960] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9391.696828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9397.421667] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9397.714335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9400.804042] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9400.811512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9400.819056] LustreError: Skipped 1 previous similar message [ 9400.820481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9400.863928] Lustre: Skipped 1 previous similar message [ 9405.927638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9405.929676] Lustre: Skipped 1 previous similar message [ 9411.049206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9411.064407] Lustre: Skipped 1 previous similar message [ 9414.112234] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9414.264887] Lustre: server umount lustre-OST0000 complete [ 9418.217562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9418.238164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9418.257502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9421.292685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9424.317253] Lustre: server umount lustre-MDT0000 complete [ 9428.543805] LustreError: 285670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390340 with bad export cookie 11698251288677652777 [ 9428.547524] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9428.563594] LustreError: 285670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9428.842740] Lustre: server umount lustre-MDT0001 complete [ 9438.703924] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 05:19:09 (1787390349) [ 9446.935397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9447.362868] LustreError: 290062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9447.377161] LustreError: 290062:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9447.428760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9452.522924] LustreError: 290063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9452.782904] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9457.644925] LustreError: 290062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9462.500128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9462.760676] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9462.773046] Lustre: Skipped 2 previous similar messages [ 9462.978664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9467.910285] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9477.554943] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9485.510430] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9494.038667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9494.456351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9501.839571] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9511.621362] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9527.267111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9527.271883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9527.274069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9527.292055] Lustre: Skipped 1 previous similar message [ 9533.003383] Lustre: server umount lustre-OST0000 complete [ 9537.505660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9537.764895] Lustre: server umount lustre-MDT0000 complete [ 9542.090323] LustreError: 290043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390454 with bad export cookie 11698251288677653659 [ 9542.091630] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9542.099207] LustreError: 290043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9544.658238] Lustre: server umount lustre-MDT0001 complete [ 9558.471159] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 9562.240729] Key type lgssc unregistered [ 9562.672959] LNet: 293623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9562.685639] LNetError: 293623:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9562.701816] LNet: Removed LNI 192.168.202.120@tcp [ 9563.836093] Key type .llcrypt unregistered [ 9563.839738] Key type ._llcrypt unregistered [ 9597.467920] Key type ._llcrypt registered [ 9597.471674] Key type .llcrypt registered [ 9597.611693] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 9616.243698] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 9617.996049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9618.121840] alg: No test for adler32 (adler32-zlib) [ 9619.331774] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 9619.604585] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 9621.297341] Key type lgssc registered [ 9622.681089] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9675.165920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9675.192125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9676.484698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9676.536879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9676.713740] Lustre: lustre-MDT0000: new disk, initializing [ 9676.822119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9676.843758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9682.653938] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9695.189795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9695.367191] Lustre: 297458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9695.469679] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9695.473826] Lustre: Skipped 1 previous similar message [ 9695.618325] Lustre: lustre-MDT0001: new disk, initializing [ 9695.748995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9695.815418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9695.821726] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9700.941549] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9711.611874] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9718.702719] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9726.794646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9727.222667] Lustre: lustre-OST2710: new disk, initializing [ 9727.225516] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9727.234286] Lustre: 299430:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9727.361627] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9734.239907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9734.257320] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9734.465661] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9736.694686] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9746.876835] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 9757.194346] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9757.411400] Lustre: lustre-OST03e8: new disk, initializing [ 9757.414486] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9757.419773] Lustre: 300495:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9757.535595] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9765.444701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9765.447506] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9765.576494] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9766.510972] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9777.501907] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 9787.375638] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9787.704507] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9793.010434] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9793.190314] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9798.957463] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9799.282833] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9805.644104] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9806.110619] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9813.473849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9813.490075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9813.511702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9817.064495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9817.069062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9817.081454] Lustre: Skipped 1 previous similar message [ 9822.183981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9822.196723] Lustre: Skipped 6 previous similar messages [ 9827.296146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9827.312570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9827.332800] Lustre: Skipped 3 previous similar messages [ 9827.514034] Lustre: server umount lustre-MDT0000 complete [ 9831.606027] LustreError: 297449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390743 with bad export cookie 11481555286164123389 [ 9831.630051] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9832.129237] Lustre: server umount lustre-MDT0001 complete [ 9845.903454] Lustre: server umount lustre-OST2710 complete [ 9847.848437] Lustre: 295254:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390744/real 1787390744] req@00000000078234cc x1874214813513216/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787390760 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9847.875399] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9847.889775] Lustre: Skipped 1 previous similar message [ 9850.656334] Lustre: server umount lustre-OST03e8 complete [ 9871.751236] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 9883.713099] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 9925.483508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9925.735813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9925.779974] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9925.840035] Lustre: lustre-MDT0000: new disk, initializing [ 9925.925164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9925.959707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9929.799808] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9943.232481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9943.469677] Lustre: 305750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9943.579494] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9943.591057] Lustre: Skipped 1 previous similar message [ 9943.693692] Lustre: lustre-MDT0001: new disk, initializing [ 9943.815149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9943.871920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9943.891103] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9949.943402] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9961.199439] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9968.324372] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9976.446234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9976.750044] Lustre: lustre-OST0000: new disk, initializing [ 9976.752680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9976.758911] Lustre: 307718:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9976.879781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9984.975860] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9986.601779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9986.614330] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9986.774139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9994.326531] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10000.188289] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10000.416559] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10005.548947] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10005.783750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10012.641384] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10012.644601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10012.650199] LustreError: Skipped 1 previous similar message [10012.653688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10012.677743] Lustre: Skipped 1 previous similar message [10022.368131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10023.071336] Lustre: server umount lustre-OST0000 complete [10031.085738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10031.101163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10031.132955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10031.149551] Lustre: Skipped 3 previous similar messages [10034.727105] Lustre: server umount lustre-MDT0000 complete [10038.242768] LustreError: 305761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10038.266932] LustreError: 305761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10038.974607] LustreError: 307717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390951 with bad export cookie 11481555286164125083 [10038.976243] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10038.980476] LustreError: 307717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10039.417677] Lustre: server umount lustre-MDT0001 complete [10048.573671] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 05:29:19 (1787390959) [10050.423543] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10052.493945] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 05:29:23 (1787390963) [10057.010094] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [10087.926675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10088.264374] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10088.279302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10088.538094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10094.818813] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10104.197135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10104.277557] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10104.311746] Lustre: 311579:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10109.932809] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10132.274983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10132.461990] Lustre: Denying initial registration attempt from nid 192.168.202.120@tcp, specified as failover [10132.471781] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10132.481241] LustreError: 313034:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [10132.494171] LustreError: 313034:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10132.506063] LustreError: 313034:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10132.645177] Lustre: server umount lustre-OST0000 complete [10132.650167] LustreError: 313034:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [10145.761818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10145.761943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10145.777689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10145.777701] Lustre: Skipped 2 previous similar messages [10145.826996] Lustre: Skipped 2 previous similar messages [10146.877937] LustreError: 313402:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (00000000b55c1cfb) refcount nonzero (2) after lock cleanup; forcing cleanup. [10147.158809] Lustre: server umount lustre-MDT0000 complete [10150.881418] LustreError: 312588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10150.908671] LustreError: 312588:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10152.783755] LustreError: 311569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391065 with bad export cookie 11481555286164125965 [10152.803264] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10152.808219] LustreError: 311569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10153.257008] Lustre: server umount lustre-MDT0001 complete [10167.328829] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [10171.182681] Key type lgssc unregistered [10171.551535] LNet: 314157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10171.566440] LNetError: 314157:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10171.590759] LNet: Removed LNI 192.168.202.120@tcp [10172.685201] Key type .llcrypt unregistered [10172.687984] Key type ._llcrypt unregistered [10184.293900] Key type ._llcrypt registered [10184.296616] Key type .llcrypt registered [10184.404696] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [10185.447572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10185.480074] alg: No test for adler32 (adler32-zlib) [10186.598825] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [10186.933494] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [10188.688189] Key type lgssc registered [10190.126431] Lustre: Echo OBD driver; http://www.lustre.org/ [10244.421161] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [10258.608402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10258.631977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10259.961982] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10259.976436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10260.115811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10265.573693] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10276.000399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10276.106202] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10276.163153] Lustre: 316934:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10276.566130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10282.525744] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10290.963356] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10298.630788] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10321.394362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10321.971475] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10322.364771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10331.260931] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10342.308866] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10345.454247] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10345.480751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10345.482840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10350.565693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10350.570484] Lustre: Skipped 2 previous similar messages [10351.359664] Lustre: server umount lustre-OST0000 complete [10358.756168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10358.774581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10358.796370] Lustre: Skipped 1 previous similar message [10358.808619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10360.802469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10360.824724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10360.829391] Lustre: Skipped 1 previous similar message [10363.235929] Lustre: server umount lustre-MDT0000 complete [10367.920261] LustreError: 316927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391280 with bad export cookie 8148812109203557369 [10367.929404] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10368.253638] Lustre: server umount lustre-MDT0001 complete [10384.692064] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [10389.757390] Key type lgssc unregistered [10390.216300] LNet: 320512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10390.231541] LNetError: 320512:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10390.276951] LNet: Removed LNI 192.168.202.120@tcp [10392.045758] Key type .llcrypt unregistered [10392.052061] Key type ._llcrypt unregistered [10423.940277] Key type ._llcrypt registered [10423.952367] Key type .llcrypt registered [10424.213350] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [10426.515487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10426.547720] alg: No test for adler32 (adler32-zlib) [10428.110953] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [10428.472285] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [10430.136165] Key type lgssc registered [10431.929881] Lustre: Echo OBD driver; http://www.lustre.org/ [10442.168128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10442.184619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10443.680447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10450.435723] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10452.971710] LustreError: 321934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10452.999630] LustreError: 321934:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10458.083734] LustreError: 321933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10461.725473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10462.383271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10468.946500] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10478.173915] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10485.404635] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10496.886229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10497.521844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10505.185658] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10513.956324] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10523.621810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10523.647389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10523.667103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10525.985840] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.20@tcp (stopping) [10525.991652] Lustre: Skipped 3 previous similar messages [10529.252457] LustreError: 321928:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10529.278735] LustreError: 321928:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10529.350078] Lustre: server umount lustre-MDT0000 complete [10534.206148] LustreError: 321915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391446 with bad export cookie 18299993157851303271 [10534.230973] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10534.370091] LustreError: 321930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10534.371630] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10534.391877] LustreError: 321930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10534.417652] Lustre: Skipped 2 previous similar messages [10534.434914] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10539.493139] LustreError: 321929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10539.495816] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10539.520588] LustreError: 321929:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10539.549807] Lustre: Skipped 1 previous similar message [10540.811426] Lustre: server umount lustre-MDT0001 complete [10548.647704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10559.051069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10579.424901] LustreError: 321242:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005e67ab49 x1874215661766784/t0(0) o250->MGC192.168.202.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10579.748168] LustreError: 323767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10579.827425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10579.910027] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10585.362639] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10595.308498] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10596.187149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10596.855721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10601.982503] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10602.174175] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10611.620665] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10613.988670] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10621.120277] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10623.080951] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10631.144232] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10631.163779] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10631.184197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10632.673971] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10636.224800] Lustre: server umount lustre-OST0000 complete [10642.913415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10642.920795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10642.937446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10642.946036] Lustre: Skipped 2 previous similar messages [10642.964208] Lustre: Skipped 3 previous similar messages [10648.046128] LustreError: 325529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10648.070515] LustreError: 325529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [10648.195812] Lustre: server umount lustre-MDT0000 complete [10653.861611] LustreError: 321915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391566 with bad export cookie 18299993157851310446 [10653.867168] LustreError: 321915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10653.870750] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10654.279818] Lustre: server umount lustre-MDT0001 complete [10667.471155] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [10671.650992] Key type lgssc unregistered [10672.018775] LNet: 328204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10672.022961] LNetError: 328204:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10672.032353] LNet: Removed LNI 192.168.202.120@tcp [10673.054332] Key type .llcrypt unregistered [10673.056125] Key type ._llcrypt unregistered [10707.564758] Key type ._llcrypt registered [10707.574780] Key type .llcrypt registered [10707.794818] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [10710.287991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10710.560538] alg: No test for adler32 (adler32-zlib) [10712.000774] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [10712.384374] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [10714.128547] Key type lgssc registered [10716.826546] Lustre: Echo OBD driver; http://www.lustre.org/ [10729.180597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10729.241044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10731.110828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10736.334572] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10741.257224] LustreError: 329627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10741.281684] LustreError: 329627:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10746.341996] LustreError: 329628:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10747.105351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10747.635907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10753.090892] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10764.441585] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10774.813851] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10778.593943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10778.604385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10778.634727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10781.665038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10781.670225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10785.052834] Lustre: server umount lustre-MDT0000 complete [10786.787294] LustreError: 330601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10786.818540] LustreError: 330601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10789.403978] LustreError: 329608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391701 with bad export cookie 7398140583450136180 [10789.407598] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10789.416648] LustreError: 329608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10789.780410] Lustre: server umount lustre-MDT0001 complete [10808.737226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10809.217460] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10809.263648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10809.516061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10814.422489] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10824.646496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10824.943233] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10825.043836] Lustre: 332216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10825.570129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10833.333137] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10844.642198] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10852.175633] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10863.409637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10863.681116] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10863.692260] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10863.703933] LustreError: 334104:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10863.711883] LustreError: 334104:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10863.718613] LustreError: 334104:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10863.799824] Lustre: server umount lustre-OST0000 complete [10863.805541] LustreError: 334104:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [10875.698130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10876.004481] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10876.367231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10885.886549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10886.157862] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10896.000172] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10909.384348] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10909.610693] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10909.808743] Lustre: lustre-OST0001: new disk, initializing [10909.820689] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10909.838780] Lustre: 335648:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10910.136572] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10914.991259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10915.009344] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10915.191581] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10918.311508] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10926.380471] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10930.664552] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10930.668599] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10930.677193] LustreError: Skipped 1 previous similar message [10930.677923] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10930.677927] Lustre: Skipped 1 previous similar message [10930.722721] Lustre: Skipped 1 previous similar message [10935.456514] Lustre: server umount lustre-OST0001 complete [10935.783591] LustreError: 334572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10935.805757] LustreError: 334572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10940.900765] LustreError: 334570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10940.913952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10940.915623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10940.915631] Lustre: Skipped 1 previous similar message [10940.917381] LustreError: 334570:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10940.948695] Lustre: Skipped 1 previous similar message [10946.022438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10946.026547] Lustre: Skipped 2 previous similar messages [10946.376688] Lustre: server umount lustre-OST0000 complete [10951.149685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10951.168427] Lustre: Skipped 1 previous similar message [10956.261260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10956.270085] Lustre: Skipped 3 previous similar messages [10956.600268] Lustre: server umount lustre-MDT0000 complete [10961.094262] LustreError: 332956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391873 with bad export cookie 7398140583450136733 [10961.107024] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10961.109215] LustreError: 332956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10961.405707] Lustre: server umount lustre-MDT0001 complete [10974.170845] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [10977.637578] Key type lgssc unregistered [10978.092401] LNet: 337604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10978.125723] LNetError: 337604:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10978.149103] LNet: Removed LNI 192.168.202.120@tcp [10979.125821] Key type .llcrypt unregistered [10979.131148] Key type ._llcrypt unregistered [11055.263851] Key type ._llcrypt registered [11055.266570] Key type .llcrypt registered [11055.370461] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [11075.520877] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [11076.852116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11076.875321] alg: No test for adler32 (adler32-zlib) [11077.914972] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [11078.229525] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [11079.984970] Key type lgssc registered [11080.975916] Lustre: Echo OBD driver; http://www.lustre.org/ [11122.129387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11122.172461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11123.627604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11123.669680] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11123.815910] Lustre: lustre-MDT0000: new disk, initializing [11123.922249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11123.940875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11129.964533] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11142.138397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11142.291672] Lustre: 342539:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11142.350615] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11142.354543] Lustre: Skipped 1 previous similar message [11142.436460] Lustre: lustre-MDT0001: new disk, initializing [11142.558110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11142.594119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11142.612405] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11148.146098] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11158.233618] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11164.075402] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11171.671561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11171.985276] Lustre: lustre-OST0000: new disk, initializing [11171.989472] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11171.997716] Lustre: 344508:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11172.138085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11179.089082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11179.105030] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11179.239951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11179.388633] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11189.649409] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11195.307513] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11195.678905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11201.513481] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11201.794465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11205.107774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11205.126373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11205.143337] LustreError: Skipped 1 previous similar message [11205.147281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11205.166841] Lustre: Skipped 1 previous similar message [11210.237353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11210.244734] Lustre: Skipped 2 previous similar messages [11210.897562] Lustre: server umount lustre-OST0000 complete [11219.432572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11219.445554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11219.462543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11220.466672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11222.929271] Lustre: server umount lustre-MDT0000 complete [11226.854212] LustreError: 342533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392139 with bad export cookie 12256266643945509481 [11226.863108] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11226.874708] LustreError: 342533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11227.277400] Lustre: server umount lustre-MDT0001 complete [11237.052316] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 05:49:07 (1787392147) [11247.950709] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 05:49:18 (1787392158) [11259.155342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11259.569826] LustreError: 347263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11259.584640] LustreError: 347263:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11259.641751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11264.996519] LustreError: 347264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11265.630240] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11269.106679] LustreError: 347263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11274.211294] LustreError: 347263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11274.984988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11275.625279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11280.688467] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11289.104086] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11295.962208] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11304.382769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11304.837306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11311.161270] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11318.944356] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11337.231764] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11339.631968] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11341.226692] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11343.238878] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11345.371027] Lustre: Failing over lustre-MDT0000 [11345.565481] Lustre: server umount lustre-MDT0000 complete [11347.429210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11347.439749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11347.455367] LustreError: 348256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11360.228159] LustreError: 348256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11360.244344] LustreError: 348256:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [11364.113582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11364.200629] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11364.399703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11364.448179] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11365.636758] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11368.545582] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11369.458527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11369.499500] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11369.619963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11379.408954] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11381.329895] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11386.004254] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11388.372928] Lustre: Failing over lustre-MDT0000 [11388.689930] Lustre: server umount lustre-MDT0000 complete [11389.922479] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11389.932793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11389.955861] Lustre: Skipped 2 previous similar messages [11389.968437] LustreError: 348256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11389.984291] LustreError: 348256:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11406.304094] Lustre: 340763:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787392302/real 1787392302] req@0000000095c573de x1874216343401984/t0(0) o400->MGC192.168.202.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787392318 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11406.319383] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11407.911754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11416.545666] LustreError: 340760:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000270cb796 x1874216343407360/t0(0) o250->MGC192.168.202.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11416.831944] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.20@tcp (not set up) [11416.841066] Lustre: Skipped 2 previous similar messages [11416.971681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11417.071845] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11421.429681] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11421.984521] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11422.205519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11422.220857] Lustre: Skipped 2 previous similar messages [11422.292453] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11422.359735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11435.610368] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11438.521063] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11447.783877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11447.790653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11447.791223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11447.830325] Lustre: Skipped 4 previous similar messages [11447.900076] Lustre: Skipped 2 previous similar messages [11452.308579] Lustre: server umount lustre-MDT0000 complete [11452.899176] LustreError: 348256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11452.939852] LustreError: 348256:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [11457.510533] LustreError: 347245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392369 with bad export cookie 12256266643945512190 [11457.524236] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11457.530171] LustreError: 347245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11457.956649] Lustre: server umount lustre-MDT0001 complete [11472.451393] Lustre: server umount lustre-OST0000 complete [11488.784276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11489.295732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11494.079963] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11502.947475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11508.651907] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11518.640950] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11525.669287] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11533.413924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11534.902852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11540.579546] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11548.600131] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11555.305877] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11555.315219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11555.359611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11555.989441] Lustre: server umount lustre-OST0000 complete [11566.751734] Lustre: server umount lustre-MDT0000 complete [11568.614446] LustreError: 355061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11568.639959] LustreError: 355061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11571.069839] LustreError: 353606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392483 with bad export cookie 12256266643945512939 [11571.075169] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11571.080458] LustreError: 353606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11581.523293] Lustre: server umount lustre-MDT0001 complete [11593.222743] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [11596.243415] Key type lgssc unregistered [11596.580630] LNet: 357162:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11596.586778] LNetError: 357162:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11597.606741] LNet: Removed LNI 192.168.202.120@tcp [11598.717843] Key type .llcrypt unregistered [11598.720217] Key type ._llcrypt unregistered [11625.435565] Key type ._llcrypt registered [11625.437593] Key type .llcrypt registered [11625.616254] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [11627.342451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11627.419127] alg: No test for adler32 (adler32-zlib) [11628.585654] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [11628.952857] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [11630.713919] Key type lgssc registered [11632.512882] Lustre: Echo OBD driver; http://www.lustre.org/ [11641.418308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11641.468815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11643.097474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11648.189925] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11653.602587] LustreError: 358680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11653.624494] LustreError: 358680:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11654.626807] LustreError: 358681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11657.748981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11658.134669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11664.096134] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11672.590165] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11679.139500] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11686.500738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11686.932428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11689.017864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11693.382783] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11701.378552] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11715.571242] Lustre: server umount lustre-MDT0000 complete [11718.629542] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11718.643534] LustreError: 358681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11718.653945] LustreError: 358681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11718.860270] LustreError: 358661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392631 with bad export cookie 17081641343069748628 [11718.865812] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11718.867937] LustreError: 358661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11721.206776] Lustre: server umount lustre-MDT0001 complete [11735.384131] Lustre: server umount lustre-OST0000 complete [11751.170470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11751.878467] LustreError: 362413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11751.991519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11757.026086] LustreError: 362412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11758.924959] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11767.284880] LustreError: 362413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11767.303847] LustreError: 362413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11769.994612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11770.541565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11776.109108] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11784.879344] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11792.099667] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11801.023901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11801.574602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11808.646326] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11817.538430] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11822.099156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11836.898090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11836.910378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11836.927334] Lustre: Skipped 2 previous similar messages [11836.942408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11842.142760] Lustre: server umount lustre-MDT0000 complete [11842.537111] LustreError: 363141:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11842.564972] LustreError: 363141:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11847.150133] LustreError: 362394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392759 with bad export cookie 17081641343069749741 [11847.164768] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11847.482871] Lustre: server umount lustre-MDT0001 complete [11862.441922] Lustre: server umount lustre-OST0000 complete [11884.135137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11884.867255] LustreError: 366328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11885.011814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11891.340088] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11903.170611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11909.907956] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11920.390400] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11927.950561] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11936.486339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11936.894536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11936.899209] Lustre: Skipped 1 previous similar message [11945.267341] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11955.316072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11956.043752] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11963.880080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11963.890304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11963.903644] Lustre: Skipped 2 previous similar messages [11963.907152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11963.915367] Lustre: Skipped 3 previous similar messages [11965.928369] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11965.934078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11965.938639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11965.973565] Lustre: Skipped 1 previous similar message [11967.393886] Lustre: server umount lustre-OST0000 complete [11975.151550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11975.167188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11975.198881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11979.623068] Lustre: server umount lustre-MDT0000 complete [11981.284096] LustreError: 366328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11981.317155] LustreError: 366328:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11984.019222] LustreError: 368176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392896 with bad export cookie 17081641343069750756 [11984.024263] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11984.034176] LustreError: 368176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11984.385712] Lustre: server umount lustre-MDT0001 complete [11998.768935] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [12001.624405] Key type lgssc unregistered [12001.962864] LNet: 369874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12001.973542] LNetError: 369874:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12001.994844] LNet: Removed LNI 192.168.202.120@tcp [12002.778193] Key type .llcrypt unregistered [12002.780075] Key type ._llcrypt unregistered [12030.481544] Key type ._llcrypt registered [12030.488484] Key type .llcrypt registered [12030.605091] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [12031.785148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12031.922018] alg: No test for adler32 (adler32-zlib) [12033.053780] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [12033.273222] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [12034.984191] Key type lgssc registered [12036.502463] Lustre: Echo OBD driver; http://www.lustre.org/ [12044.845297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12044.860649] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12044.868058] LustreError: 371314:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [12044.883178] LustreError: 371314:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [12044.887873] LustreError: 371314:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [12044.900150] LustreError: 371314:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [12044.910458] LustreError: 371314:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [12054.561417] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [12054.577197] LustreError: 371714:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [12054.613839] LustreError: 371714:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [12054.618443] LustreError: 371714:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [12054.631433] LustreError: 371714:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [12054.644656] LustreError: 371714:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [12077.574914] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [12080.362894] Key type lgssc unregistered [12080.692218] LNet: 372716:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12080.697712] LNetError: 372716:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12080.722906] LNet: Removed LNI 192.168.202.120@tcp [12081.669453] Key type .llcrypt unregistered [12081.671137] Key type ._llcrypt unregistered [12104.479260] Key type ._llcrypt registered [12104.481496] Key type .llcrypt registered [12104.589743] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [12123.991662] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [12125.902828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12126.023444] alg: No test for adler32 (adler32-zlib) [12127.047338] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [12127.430645] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [12129.200204] Key type lgssc registered [12130.750788] Lustre: Echo OBD driver; http://www.lustre.org/ [12175.788473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12175.830760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12177.136451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12177.193382] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12177.253464] Lustre: lustre-MDT0000: new disk, initializing [12177.292230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12177.309815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12182.124351] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12195.746899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12195.831515] Lustre: 375879:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12195.879832] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12195.884135] Lustre: Skipped 1 previous similar message [12195.947342] Lustre: lustre-MDT0001: new disk, initializing [12196.046920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12196.123555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12196.140097] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12202.760682] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12216.018520] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12224.869201] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12234.092236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12234.429580] Lustre: lustre-OST0000: new disk, initializing [12234.432941] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12234.443734] Lustre: 377849:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12234.578455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12241.288718] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12241.497797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12241.501315] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12241.673197] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12251.991938] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12257.243838] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12257.579518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12262.304443] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12262.529482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12267.493329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12267.499631] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12267.500742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12267.550100] Lustre: Skipped 1 previous similar message [12272.622468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12272.639103] Lustre: Skipped 1 previous similar message [12277.774894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12277.788934] Lustre: Skipped 2 previous similar messages [12279.268983] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12279.681270] Lustre: server umount lustre-OST0000 complete [12287.969577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12287.973202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12287.986490] Lustre: Skipped 1 previous similar message [12291.259669] Lustre: server umount lustre-MDT0000 complete [12295.220129] LustreError: 375870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393207 with bad export cookie 16345416761044111381 [12295.221086] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12295.227967] LustreError: 375870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12295.859847] Lustre: server umount lustre-MDT0001 complete [12307.574125] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12309.714398] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 06:07:00 (1787393220) [12317.284237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12318.277123] LustreError: 380311:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12318.301256] LustreError: 380311:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12318.389732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12323.815909] LustreError: 380312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12323.915202] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12327.912644] LustreError: 380311:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12333.029756] LustreError: 380312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12336.444486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12336.994078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12342.154390] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12353.325939] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12361.457656] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12369.301030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12369.695404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12376.971651] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12385.431467] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12394.014067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12394.292345] Lustre: lustre-OST0001: new disk, initializing [12394.297258] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12394.307336] Lustre: 383152:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12394.429204] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12399.751530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12399.763734] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12399.981177] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12402.208311] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12413.639922] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [12420.577321] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12420.582544] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12420.588835] LustreError: Skipped 2 previous similar messages [12420.589844] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12420.589849] Lustre: Skipped 1 previous similar message [12420.627248] Lustre: Skipped 1 previous similar message [12424.299358] Lustre: server umount lustre-OST0001 complete [12425.705662] LustreError: 382136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12425.723905] LustreError: 382136:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12435.942903] LustreError: 383386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12435.967046] LustreError: 383386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12443.618181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12443.647512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12443.680323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12443.688409] Lustre: Skipped 2 previous similar messages [12448.847412] Lustre: server umount lustre-OST0000 complete [12454.881794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12454.893330] LustreError: Skipped 1 previous similar message [12454.896608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12454.906913] Lustre: Skipped 1 previous similar message [12461.538740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12461.541420] Lustre: Skipped 6 previous similar messages [12468.192491] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12468.366245] Lustre: server umount lustre-MDT0000 complete [12471.779484] LustreError: 380307:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12471.796746] LustreError: 380307:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12472.967426] LustreError: 380292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393385 with bad export cookie 16345416761044112263 [12472.980238] LustreError: 380292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12472.984572] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12479.652787] Lustre: server umount lustre-MDT0001 complete [12490.791955] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [12493.407918] Key type lgssc unregistered [12493.774754] LNet: 385143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12493.784125] LNetError: 385143:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12494.823275] LNet: Removed LNI 192.168.202.120@tcp [12496.430819] Key type .llcrypt unregistered [12496.433083] Key type ._llcrypt unregistered [12557.873550] Key type ._llcrypt registered [12557.881033] Key type .llcrypt registered [12558.083328] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [12559.808348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12560.004710] alg: No test for adler32 (adler32-zlib) [12561.172357] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [12561.619040] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [12563.440958] Key type lgssc registered [12565.113574] Lustre: Echo OBD driver; http://www.lustre.org/ [12575.410192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12575.431765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12576.761841] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12576.802344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12577.064466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12581.693837] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12593.326660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12593.388671] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12593.414314] Lustre: 387943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12593.596525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12601.560810] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12611.497745] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12618.208951] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12622.820245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12622.826832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12622.849607] Lustre: Skipped 1 previous similar message [12622.877183] Lustre: Skipped 1 previous similar message [12627.938038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12627.946897] Lustre: Skipped 1 previous similar message [12633.060739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12633.067507] Lustre: Skipped 1 previous similar message [12635.104235] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12636.025853] Lustre: server umount lustre-MDT0000 complete [12641.166623] LustreError: 387935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393553 with bad export cookie 9404824547737345645 [12641.197899] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12641.210043] LustreError: 387935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12641.734459] Lustre: server umount lustre-MDT0001 complete [12654.601251] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 06:12:45 (1787393565) [12662.933688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12663.565328] LustreError: 390457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12663.582187] LustreError: 390457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12663.656092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12668.901188] LustreError: 390458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12670.088577] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12674.022880] LustreError: 390457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12679.145867] LustreError: 390458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12680.722655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12681.244945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12686.632796] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12696.249913] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12703.192987] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12711.572350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12711.842185] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12712.151199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12719.550704] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12729.007158] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12737.860521] Lustre: Permanently deactivating lustre-OST0000 [12737.887526] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12745.751480] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12748.037563] LustreError: 393447:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12748.045031] LustreError: 393447:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12748.053674] LustreError: 393447:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12751.861807] LustreError: 393497:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12751.864812] LustreError: 393497:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12761.043562] Lustre: server umount lustre-OST0000 complete [12764.424826] LustreError: 393748:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12764.441186] LustreError: 393748:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12767.714670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12767.719140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12767.738234] Lustre: Skipped 1 previous similar message [12767.753225] Lustre: Skipped 1 previous similar message [12772.834486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12772.852148] Lustre: Skipped 1 previous similar message [12777.144955] Lustre: server umount lustre-MDT0000 complete [12777.953302] LustreError: 390453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12777.966732] LustreError: 390453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12782.071601] LustreError: 390437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393694 with bad export cookie 9404824547737346219 [12782.081414] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12782.086807] LustreError: 390437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12792.574954] Lustre: server umount lustre-MDT0001 complete [12802.324498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12807.870849] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12811.441591] LustreError: 394907:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12818.948270] Lustre: 395246:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.120@tcp [12821.428110] Lustre: server umount MGS complete [12834.123541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12834.461931] LustreError: 395970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12834.496542] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12834.501910] Lustre: Skipped 1 previous similar message [12834.569277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12840.670214] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12851.723720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12852.079223] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12852.170711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12857.335746] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12865.851608] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12872.609578] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12875.084722] Lustre: Permanently reactivating lustre-OST0000 [12875.112080] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12875.118858] Lustre: Skipped 2 previous similar messages [12902.880885] LustreError: 397703:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12914.864685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12915.379939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12922.416994] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12930.474840] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12937.745080] Lustre: server umount lustre-OST0000 complete [12944.354147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12944.355775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12944.393223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12944.408266] Lustre: Skipped 1 previous similar message [12952.625660] Lustre: server umount lustre-MDT0000 complete [12954.598285] LustreError: 395966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12954.628778] LustreError: 395966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12955.616665] Lustre: 387266:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787393851/real 1787393851] req@000000004c5098e5 x1874217898448384/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787393867 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12955.650290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12955.676556] Lustre: Skipped 1 previous similar message [12956.743445] LustreError: 395950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393868 with bad export cookie 9404824547737347276 [12956.744608] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12956.751482] LustreError: 395950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12957.309491] Lustre: server umount lustre-MDT0001 complete [12968.197141] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [12971.432825] Key type lgssc unregistered [12971.700655] LNet: 400241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12971.704484] LNetError: 400241:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12971.721274] LNet: Removed LNI 192.168.202.120@tcp [12972.643914] Key type .llcrypt unregistered [12972.648327] Key type ._llcrypt unregistered [12992.484281] Key type ._llcrypt registered [12992.488547] Key type .llcrypt registered [12992.667916] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [13008.780180] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [13010.223262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13010.401080] alg: No test for adler32 (adler32-zlib) [13011.481901] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [13011.737751] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [13013.473090] Key type lgssc registered [13014.985738] Lustre: Echo OBD driver; http://www.lustre.org/ [13053.983994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13054.024093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13055.350888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13055.386339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13055.473756] Lustre: lustre-MDT0000: new disk, initializing [13055.589698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13055.610354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13060.379431] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13071.340285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13071.549490] Lustre: 403401:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13071.650945] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13071.653949] Lustre: Skipped 1 previous similar message [13071.757778] Lustre: lustre-MDT0001: new disk, initializing [13071.947672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13072.010861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13072.023776] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13076.606533] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13087.173293] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13093.674601] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13100.945794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13101.188858] Lustre: lustre-OST0000: new disk, initializing [13101.193537] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13101.201438] Lustre: 405372:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13101.295656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13107.072569] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13109.904777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13109.935568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13110.247909] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13118.428287] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13125.118167] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13125.386745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13130.315447] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13130.682285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13135.852810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13135.863635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13135.877820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13135.880789] LustreError: Skipped 1 previous similar message [13139.762135] Lustre: server umount lustre-OST0000 complete [13146.082072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13146.105044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13146.107555] Lustre: Skipped 2 previous similar messages [13146.140624] Lustre: Skipped 2 previous similar messages [13150.453698] Lustre: server umount lustre-MDT0000 complete [13154.494504] LustreError: 403395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394066 with bad export cookie 15079891992322235588 [13154.513691] LustreError: 403395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13154.515068] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13154.910774] Lustre: server umount lustre-MDT0001 complete [13165.412758] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13167.102473] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 06:21:18 (1787394078) [13173.689548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13174.142772] LustreError: 407832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13174.168175] LustreError: 407832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13174.225928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13179.022313] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13179.371929] LustreError: 407833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13184.484943] LustreError: 407832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13189.604308] LustreError: 407833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13190.015106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13190.717519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13196.719687] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13207.745928] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13214.870851] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13221.750668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13222.172954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13229.415682] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13237.143914] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13239.343755] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13246.016919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13255.140911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13255.146425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13255.158180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13255.166947] Lustre: Skipped 1 previous similar message [13255.184124] Lustre: Skipped 1 previous similar message [13257.813483] Lustre: server umount lustre-OST0000 complete [13265.378036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13265.381759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13265.395130] Lustre: Skipped 1 previous similar message [13265.405699] Lustre: Skipped 1 previous similar message [13268.869391] Lustre: server umount lustre-MDT0000 complete [13270.505995] LustreError: 407827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13270.532179] LustreError: 407827:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13272.787197] LustreError: 407814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394185 with bad export cookie 15079891992322236470 [13272.821683] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13272.824103] LustreError: 407814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13273.118503] Lustre: server umount lustre-MDT0001 complete [13285.055719] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [13288.068598] Key type lgssc unregistered [13288.430247] LNet: 411559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13288.452924] LNetError: 411559:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13288.477405] LNet: Removed LNI 192.168.202.120@tcp [13289.570857] Key type .llcrypt unregistered [13289.576500] Key type ._llcrypt unregistered [13317.905744] Key type ._llcrypt registered [13317.923319] Key type .llcrypt registered [13318.134550] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [13318.599545] Key type .llcrypt unregistered [13318.602781] Key type ._llcrypt unregistered [13340.242054] Key type ._llcrypt registered [13340.245566] Key type .llcrypt registered [13340.350994] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [13341.859968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13341.914543] alg: No test for adler32 (adler32-zlib) [13343.032972] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [13343.225407] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [13344.897725] Key type lgssc registered [13346.506217] Lustre: Echo OBD driver; http://www.lustre.org/ [13354.697153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13354.713812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13356.173685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13362.755149] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13366.760731] LustreError: 413777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13366.794936] LustreError: 413777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13371.899161] LustreError: 413776:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13374.578868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13375.021074] LustreError: 413777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13375.042273] LustreError: 413777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13375.051366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13380.075525] LustreError: 413776:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13380.102314] LustreError: 413776:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13381.124873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13382.396393] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13386.220826] LustreError: 413777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13386.241337] LustreError: 413777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13391.990254] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13396.454976] LustreError: 413776:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13396.483637] LustreError: 413776:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13400.712808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13401.291752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13406.716423] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13416.417956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13416.433355] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13416.453260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13416.952980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13416.966060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13416.984500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13422.092435] Lustre: server umount lustre-OST0000 complete [13430.362558] Lustre: server umount lustre-MDT0000 complete [13431.794257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13431.811172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13431.831464] LustreError: 413776:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13435.205156] LustreError: 413756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394347 with bad export cookie 8545227946637492849 [13435.206567] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13435.237692] LustreError: 413756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13435.532963] Lustre: server umount lustre-MDT0001 complete [13446.734024] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [13451.386194] Key type lgssc unregistered [13451.866536] LNet: 416927:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13451.874382] LNetError: 416927:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13451.899311] LNet: Removed LNI 192.168.202.120@tcp [13452.949832] Key type .llcrypt unregistered [13452.952175] Key type ._llcrypt unregistered [13480.726087] Key type ._llcrypt registered [13480.728302] Key type .llcrypt registered [13480.828615] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [13482.035363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13482.056946] alg: No test for adler32 (adler32-zlib) [13483.185354] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [13483.577095] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [13485.264175] Key type lgssc registered [13486.866858] Lustre: Echo OBD driver; http://www.lustre.org/ [13495.717800] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13495.733516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13511.264473] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13516.384356] LustreError: 418334:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.120@tcp: failed processing log, type 4: rc = -110 [13543.073039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13550.314487] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13558.157882] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13565.319945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13565.816661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13565.916686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13571.052947] LustreError: 418359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13571.090071] LustreError: 418359:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13571.150855] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13576.172237] LustreError: 418360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13578.226446] LustreError: 419370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13578.256415] LustreError: 419370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13580.329424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13580.798706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13586.041830] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13594.930122] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13602.201575] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13611.503661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13611.521954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13611.524602] Lustre: Skipped 1 previous similar message [13611.534440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13611.535129] Lustre: Skipped 1 previous similar message [13616.554927] Lustre: server umount lustre-OST0000 complete [13621.731156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13621.735529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13621.758562] Lustre: Skipped 1 previous similar message [13621.769995] Lustre: Skipped 1 previous similar message [13623.576163] Lustre: server umount lustre-MDT0000 complete [13626.849181] LustreError: 419374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13626.866198] LustreError: 419374:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13628.116346] LustreError: 418342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394540 with bad export cookie 16716776721121171518 [13628.118377] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13628.126111] LustreError: 418342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13628.470346] Lustre: server umount lustre-MDT0001 complete [13639.389101] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [13641.527563] Key type lgssc unregistered [13641.814231] LNet: 421905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13641.819523] LNetError: 421905:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13642.857167] LNet: Removed LNI 192.168.202.120@tcp [13643.770890] Key type .llcrypt unregistered [13643.773838] Key type ._llcrypt unregistered [13673.213613] Key type ._llcrypt registered [13673.215605] Key type .llcrypt registered [13673.359847] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [13675.505616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13675.563420] alg: No test for adler32 (adler32-zlib) [13676.787922] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [13677.157642] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [13678.872728] Key type lgssc registered [13680.456083] Lustre: Echo OBD driver; http://www.lustre.org/ [13689.658632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13689.698087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13691.568658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13697.862951] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13701.093725] LustreError: 423324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13701.142798] LustreError: 423324:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13706.214164] LustreError: 423323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13708.421304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13708.952785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13714.048740] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13723.252373] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13729.725234] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13738.325903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13738.969828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13747.282381] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13755.042191] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13761.597073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13770.220493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13770.234349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13770.248258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13771.035519] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.20@tcp (stopping) [13771.746106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13771.779719] Lustre: Skipped 1 previous similar message [13775.095527] Lustre: server umount lustre-MDT0000 complete [13776.118894] LustreError: 423319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13778.447618] LustreError: 425169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.20@tcp arrived at 1787394690 with bad export cookie 7367462717349356390 [13778.473349] LustreError: 425169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13780.789880] LustreError: 423304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394693 with bad export cookie 7367462717349355718 [13780.812282] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13780.815225] LustreError: 423304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13781.985647] LustreError: 423323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13782.001123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13782.013106] LustreError: 423323:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13782.057793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13782.068279] Lustre: Skipped 3 previous similar messages [13786.019401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13786.021032] LustreError: 425166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13786.052703] LustreError: 425166:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13787.228232] Lustre: server umount lustre-OST0000 complete [13795.396275] Lustre: server umount lustre-MDT0001 complete [13810.813629] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [13814.998480] Key type lgssc unregistered [13815.340567] LNet: 427031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13815.366172] LNetError: 427031:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13815.398857] LNet: Removed LNI 192.168.202.120@tcp [13816.374969] Key type .llcrypt unregistered [13816.376849] Key type ._llcrypt unregistered [13844.061359] Key type ._llcrypt registered [13844.062749] Key type .llcrypt registered [13844.173518] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [13845.003097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13845.016014] alg: No test for adler32 (adler32-zlib) [13846.431229] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [13846.715581] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [13848.464266] Key type lgssc registered [13849.757308] Lustre: Echo OBD driver; http://www.lustre.org/ [13858.390160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13858.432579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13860.112927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13864.517429] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13870.573535] LustreError: 428446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13870.610167] LustreError: 428446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13875.298228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13875.697599] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13875.857041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13880.760821] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13890.969791] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13897.595919] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13905.404728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13905.759653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13912.806665] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13919.786696] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13930.367096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13935.588255] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13935.608607] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13935.613330] Lustre: Skipped 1 previous similar message [13936.943135] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.20@tcp (stopping) [13940.706743] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13940.727031] Lustre: Skipped 1 previous similar message [13941.000910] Lustre: server umount lustre-MDT0001 complete [13942.021389] LustreError: 428442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13945.826506] LustreError: 429629:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13945.847899] LustreError: 429629:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13950.946483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13950.946471] LustreError: 430937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13950.953128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13950.955305] Lustre: Skipped 1 previous similar message [13950.957630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13954.531945] Lustre: server umount lustre-OST0000 complete [13956.065758] LustreError: 428446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13959.072504] Lustre: server umount lustre-MDT0000 complete [13974.355285] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [13977.815860] Key type lgssc unregistered [13978.157890] LNet: 432155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13978.167506] LNetError: 432155:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13978.181279] LNet: Removed LNI 192.168.202.120@tcp [13979.020721] Key type .llcrypt unregistered [13979.023424] Key type ._llcrypt unregistered [14009.818911] Key type ._llcrypt registered [14009.825044] Key type .llcrypt registered [14009.916469] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [14026.858456] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [14028.408964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14028.449545] alg: No test for adler32 (adler32-zlib) [14029.843613] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [14030.108607] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [14031.794765] Key type lgssc registered [14033.246130] Lustre: Echo OBD driver; http://www.lustre.org/ [14080.528500] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [14099.468094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14099.489604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14100.956515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14101.032578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14101.130737] Lustre: lustre-MDT0000: new disk, initializing [14101.291813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14101.346948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14106.509956] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [14123.676996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14124.111681] Lustre: lustre-OST0000: new disk, initializing [14124.118934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14124.123442] Lustre: Skipped 1 previous similar message [14124.133566] Lustre: 437191:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14124.246664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14131.880817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14131.890660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14131.932647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14132.800160] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [14142.055226] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14163.994384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14164.179862] Lustre: 436172:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14164.272501] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14164.433447] Lustre: lustre-MDT0001: new disk, initializing [14164.648213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14164.726350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [14164.769405] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [14169.468769] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [14179.001337] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [14179.160269] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14184.654766] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14184.867260] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14192.100685] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14192.123340] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14192.130439] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14192.142590] Lustre: Skipped 1 previous similar message [14195.484056] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.20@tcp (stopping) [14197.236305] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14197.246920] Lustre: Skipped 1 previous similar message [14197.575101] Lustre: server umount lustre-MDT0001 complete [14207.465135] LustreError: 436181:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14207.483675] LustreError: 436181:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [14212.604358] LustreError: 436184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14212.645983] LustreError: 436184:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14217.698302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14217.700715] LustreError: 436185:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14217.716095] Lustre: Skipped 1 previous similar message [14217.717799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14222.817920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14222.847266] LustreError: 436184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14223.813976] Lustre: server umount lustre-OST0000 complete [14227.958784] LustreError: 436185:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14234.919328] Lustre: server umount lustre-MDT0000 complete [14250.007721] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [14253.556442] Key type lgssc unregistered [14253.900983] LNet: 440941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14253.921440] LNetError: 440941:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14254.953201] LNet: Removed LNI 192.168.202.120@tcp [14255.949598] Key type .llcrypt unregistered [14255.951282] Key type ._llcrypt unregistered [14309.491698] Key type ._llcrypt registered [14309.499953] Key type .llcrypt registered [14309.699111] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [14310.080312] Key type .llcrypt unregistered [14310.082061] Key type ._llcrypt unregistered [14329.888926] Key type ._llcrypt registered [14329.896991] Key type .llcrypt registered [14330.104260] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [14331.604133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14331.903902] alg: No test for adler32 (adler32-zlib) [14333.133112] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [14333.591312] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [14335.352238] Key type lgssc registered [14336.938507] Lustre: Echo OBD driver; http://www.lustre.org/ [14372.127755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14372.218789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14373.670158] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14373.721757] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14373.849541] Lustre: lustre-MDT0000: new disk, initializing [14373.947310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14373.973571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14379.247953] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [14391.811509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14391.926544] Lustre: 444324:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14391.980344] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14391.988694] Lustre: Skipped 1 previous similar message [14392.131994] Lustre: lustre-MDT0001: new disk, initializing [14392.212631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14392.251819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14392.257084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14399.759298] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [14413.728763] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14423.158797] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14431.068086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14431.512658] Lustre: lustre-OST0000: new disk, initializing [14431.514861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14431.521021] Lustre: 446294:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14431.660918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14439.149734] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [14440.520957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14440.543593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14440.621874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14449.803872] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14458.853338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14458.860940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14458.884244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14461.417472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14461.417518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14463.600824] Lustre: server umount lustre-MDT0000 complete [14469.022336] LustreError: 446304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395381 with bad export cookie 12489033213499705889 [14469.046080] LustreError: 446304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14469.046472] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14469.531945] Lustre: server umount lustre-MDT0001 complete [14483.233798] Lustre: server umount lustre-OST0000 complete [14495.999259] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 06:43:26 (1787395406) [14504.783791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14505.161572] LustreError: 448327:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14505.183568] LustreError: 448327:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [14505.252287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14510.568291] LustreError: 448328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14510.849324] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [14515.696154] LustreError: 448327:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14519.992834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14520.350201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14524.425741] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [14532.636861] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14539.707762] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14560.127827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14560.415064] Lustre: Found index 0 for lustre-OST0000, updating log [14560.797839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14568.271970] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [14576.184300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14577.259632] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14585.314436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14585.324682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14585.336453] Lustre: Skipped 1 previous similar message [14585.342690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14585.347299] Lustre: Skipped 2 previous similar messages [14589.590960] Lustre: server umount lustre-OST0000 complete [14607.330146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14607.341549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14607.367109] Lustre: Skipped 1 previous similar message [14607.378405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14607.389171] Lustre: Skipped 3 previous similar messages [14611.945264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14611.952714] Lustre: Skipped 2 previous similar messages [14611.988206] Lustre: server umount lustre-MDT0000 complete [14612.977349] LustreError: 448323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14613.008641] LustreError: 448323:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14615.958757] LustreError: 448308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395528 with bad export cookie 12489033213499707282 [14615.961962] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14615.969963] LustreError: 448308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14616.256516] Lustre: server umount lustre-MDT0001 complete [14621.472205] LNet: 1 local NIs in recovery (showing 1): 192.168.202.120@tcp [14623.855730] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 06:45:35 (1787395535) [15250.672513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15251.191226] LustreError: 454741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15251.311187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15256.538946] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [15256.546724] LustreError: 454742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15261.678303] LustreError: 454741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15266.948352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15267.362507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15272.965067] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [15281.851364] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15288.840452] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15309.868096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15310.145756] Lustre: Found index 0 for lustre-OST0000, updating log [15310.499973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15313.773960] LNet: 1 local NIs in recovery (showing 1): 192.168.202.120@tcp [15318.001585] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [15326.765273] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15329.845168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15329.865197] Lustre: Skipped 1 previous similar message [15329.875320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15343.074172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15343.228471] Lustre: server umount lustre-OST0000 complete [15347.296390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15347.300199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15347.305546] Lustre: Skipped 1 previous similar message [15347.318154] Lustre: Skipped 1 previous similar message [15349.347096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15349.349787] Lustre: Skipped 2 previous similar messages [15353.443515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15353.457645] Lustre: Skipped 8 previous similar messages [15353.804952] Lustre: server umount lustre-MDT0000 complete [15354.471565] LustreError: 457184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15354.512674] LustreError: 457184:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [15357.867693] LustreError: 456605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396270 with bad export cookie 12489033213499708367 [15357.871047] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15357.882411] LustreError: 456605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15358.490848] Lustre: server umount lustre-MDT0001 complete [15374.176143] LNet: 1 local NIs in recovery (showing 1): 192.168.202.120@tcp [15396.264274] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [15407.772174] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [15449.235767] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 06:59:20 (1787396360) [15469.590856] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [15474.506315] Key type lgssc unregistered [15475.139951] LNet: 461362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15475.159177] LNetError: 461362:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15570.406734] LNet: Removed LNI 192.168.202.120@tcp [15571.319651] Key type .llcrypt unregistered [15571.321337] Key type ._llcrypt unregistered [15590.252819] Key type ._llcrypt registered [15590.257658] Key type .llcrypt registered [15590.382180] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [15592.148797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15592.211855] alg: No test for adler32 (adler32-zlib) [15593.306984] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [15593.687599] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [15595.384194] Key type lgssc registered [15596.844678] Lustre: Echo OBD driver; http://www.lustre.org/ [15602.784344] LNet: Added LNI 192.168.202.120@tcp100 [8/256/0/180] [15604.056692] LNet: Added LNI 192.168.202.120@tcp101 [8/256/0/180] [15605.176360] LNet: Added LNI 192.168.202.120@tcp102 [8/256/0/180] [15606.248633] LNet: Added LNI 192.168.202.120@tcp103 [8/256/0/180] [15607.048691] LNet: Added LNI 192.168.202.120@tcp104 [8/256/0/180] [15608.062920] LNet: Added LNI 192.168.202.120@tcp105 [8/256/0/180] [15609.139541] LNet: Added LNI 192.168.202.120@tcp106 [8/256/0/180] [15610.335425] LNet: Added LNI 192.168.202.120@tcp107 [8/256/0/180] [15611.293929] LNet: Added LNI 192.168.202.120@tcp108 [8/256/0/180] [15612.444819] LNet: Added LNI 192.168.202.120@tcp109 [8/256/0/180] [15613.589148] LNet: Added LNI 192.168.202.120@tcp110 [8/256/0/180] [15614.755658] LNet: Added LNI 192.168.202.120@tcp111 [8/256/0/180] [15615.737053] LNet: Added LNI 192.168.202.120@tcp112 [8/256/0/180] [15616.770201] LNet: Added LNI 192.168.202.120@tcp113 [8/256/0/180] [15617.988026] LNet: Added LNI 192.168.202.120@tcp114 [8/256/0/180] [15619.095539] LNet: Added LNI 192.168.202.120@tcp115 [8/256/0/180] [15620.057588] LNet: Added LNI 192.168.202.120@tcp116 [8/256/0/180] [15621.128267] LNet: Added LNI 192.168.202.120@tcp117 [8/256/0/180] [15622.095341] LNet: Added LNI 192.168.202.120@tcp118 [8/256/0/180] [15623.143506] LNet: Added LNI 192.168.202.120@tcp119 [8/256/0/180] [15624.268781] LNet: Added LNI 192.168.202.120@tcp120 [8/256/0/180] [15625.382209] LNet: Added LNI 192.168.202.120@tcp121 [8/256/0/180] [15626.509724] LNet: Added LNI 192.168.202.120@tcp122 [8/256/0/180] [15627.672741] LNet: Added LNI 192.168.202.120@tcp123 [8/256/0/180] [15628.679281] LNet: Added LNI 192.168.202.120@tcp124 [8/256/0/180] [15629.759865] LNet: Added LNI 192.168.202.120@tcp125 [8/256/0/180] [15630.921816] LNet: Added LNI 192.168.202.120@tcp126 [8/256/0/180] [15632.022794] LNet: Added LNI 192.168.202.120@tcp127 [8/256/0/180] [15633.129863] LNet: Added LNI 192.168.202.120@tcp128 [8/256/0/180] [15634.461820] LNet: Added LNI 192.168.202.120@tcp129 [8/256/0/180] [15635.656518] LNet: Added LNI 192.168.202.120@tcp130 [8/256/0/180] [15636.668662] LNet: Added LNI 192.168.202.120@tcp131 [8/256/0/180] [15637.797501] LNet: Added LNI 192.168.202.120@tcp132 [8/256/0/180] [15639.076127] LNet: Added LNI 192.168.202.120@tcp133 [8/256/0/180] [15640.115360] LNet: Added LNI 192.168.202.120@tcp134 [8/256/0/180] [15641.134276] LNet: Added LNI 192.168.202.120@tcp135 [8/256/0/180] [15648.908088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15648.933963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15650.079466] Lustre: 464402:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.120@tcp: can't add NID 192.168.202.120@tcp130: rc = -75 [15650.268917] LNetError: 464428:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.120 [15650.287638] LNetError: Protocol error connecting to 192.168.202.120@tcp119 at host 192.168.202.120:988: is it running a compatible version of Lustre? [15650.313387] LNetError: Skipped 2 previous similar messages [15650.570573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15650.624342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15650.781584] Lustre: lustre-MDT0000: new disk, initializing [15650.933363] Lustre: 464457:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.120@tcp: can't add NID 192.168.202.120@tcp130: rc = -75 [15650.943089] Lustre: 464457:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15650.969974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15651.002102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15651.040443] LNet: 4 local NIs in recovery (showing 4): 192.168.202.120@tcp124, 192.168.202.120@tcp125, 192.168.202.120@tcp129, 192.168.202.120@tcp119 [15651.051652] LNetError: 464428:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.120 [15651.081528] LNetError: 464428:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [15651.105134] LNetError: Protocol error connecting to 192.168.202.120@tcp101 at host 192.168.202.120:988: is it running a compatible version of Lustre? [15651.126415] LNetError: Skipped 6 previous similar messages [15657.048165] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [15670.437774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15670.631841] Lustre: 464417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15670.719631] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15670.726722] Lustre: Skipped 1 previous similar message [15670.874797] Lustre: lustre-MDT0001: new disk, initializing [15670.938285] Lustre: 465220:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.120@tcp: can't add NID 192.168.202.120@tcp130: rc = -75 [15670.952830] Lustre: 465220:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15671.160425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15671.218277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15671.231548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15676.865553] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [15679.537615] Lustre: 465583:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.120@tcp: can't add NID 192.168.202.120@tcp130: rc = -75 [15679.549148] Lustre: 465583:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15680.869319] LNetError: 461861:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.120 [15680.902794] LNetError: Protocol error connecting to 192.168.202.120@tcp120 at host 192.168.202.120:988: is it running a compatible version of Lustre? [15682.919386] LNetError: 461860:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.120 [15682.937636] LNetError: 461860:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [15682.958120] LNetError: Protocol error connecting to 192.168.202.120@tcp128 at host 192.168.202.120:988: is it running a compatible version of Lustre? [15682.970776] LNetError: Skipped 1 previous similar message [15689.060474] LNetError: 461859:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.120 [15689.081394] LNetError: 461859:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [15689.095508] LNetError: Protocol error connecting to 192.168.202.120@tcp127 at host 192.168.202.120:988: is it running a compatible version of Lustre? [15689.113426] LNetError: Skipped 1 previous similar message [15689.394131] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15696.590173] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15701.359164] LNetError: 464431:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.120 [15701.385640] LNetError: 464431:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [15701.406844] LNetError: Protocol error connecting to 192.168.202.120@tcp127 at host 192.168.202.120:988: is it running a compatible version of Lustre? [15701.427508] LNetError: Skipped 6 previous similar messages [15706.442849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15707.004329] Lustre: lustre-OST0000: new disk, initializing [15707.008743] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15707.015931] Lustre: 466395:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15707.173748] Lustre: 466400:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.120@tcp: can't add NID 192.168.202.120@tcp130: rc = -75 [15707.188773] Lustre: 466400:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15707.266848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15717.012229] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [15717.441860] Lustre: 466765:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.120@tcp: can't add NID 192.168.202.120@tcp130: rc = -75 [15717.454747] Lustre: 466765:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15717.538920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15717.557592] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15717.740643] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15718.759321] LNetError: 464431:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.120 [15718.771582] LNetError: 464431:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [15718.785576] LNetError: Protocol error connecting to 192.168.202.120@tcp120 at host 192.168.202.120:988: is it running a compatible version of Lustre? [15718.799142] LNetError: Skipped 9 previous similar messages [15730.637559] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15743.457812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15743.468900] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15743.473415] LustreError: Skipped 1 previous similar message [15743.486620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15745.850616] Lustre: server umount lustre-OST0000 complete [15753.186417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15753.212545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15753.227221] Lustre: Skipped 1 previous similar message [15753.240392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15753.253544] Lustre: Skipped 1 previous similar message [15753.577867] LNetError: 464421:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.120 [15753.599695] LNetError: 464421:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [15753.610924] LNetError: Protocol error connecting to 192.168.202.120@tcp127 at host 192.168.202.120:988: is it running a compatible version of Lustre? [15753.622776] LNetError: Skipped 10 previous similar messages [15757.026513] Lustre: server umount lustre-MDT0000 complete [15761.668744] LustreError: 464410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396673 with bad export cookie 7884063148183204417 [15761.675899] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15761.683477] LustreError: 464410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15762.184106] Lustre: server umount lustre-MDT0001 complete [15775.996909] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [15779.916687] Key type lgssc unregistered [15780.539160] LNet: 468312:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15780.547547] LNetError: 468312:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15781.600698] LNet: Removed LNI 192.168.202.120@tcp [15781.608076] LNetError: 468312:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15781.621640] LNet: Removed LNI 192.168.202.120@tcp100 [15782.624303] LNet: Removed LNI 192.168.202.120@tcp101 [15782.636181] LNetError: 468312:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15782.652232] LNetError: 468312:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15782.662695] LNet: Removed LNI 192.168.202.120@tcp102 [15783.714073] LNet: Removed LNI 192.168.202.120@tcp103 [15784.738982] LNet: Removed LNI 192.168.202.120@tcp104 [15784.743347] LNetError: 468312:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15784.749373] LNetError: 468312:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15784.758295] LNet: Removed LNI 192.168.202.120@tcp105 [15785.824183] LNet: Removed LNI 192.168.202.120@tcp106 [15786.848170] LNet: Removed LNI 192.168.202.120@tcp107 [15787.872926] LNet: Removed LNI 192.168.202.120@tcp108 [15788.896176] LNet: Removed LNI 192.168.202.120@tcp109 [15788.898134] LNetError: 468312:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15788.910203] LNetError: 468312:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15788.930049] LNet: Removed LNI 192.168.202.120@tcp110 [15789.987329] LNet: Removed LNI 192.168.202.120@tcp111 [15791.009297] LNet: Removed LNI 192.168.202.120@tcp112 [15792.033052] LNet: Removed LNI 192.168.202.120@tcp113 [15793.056168] LNet: Removed LNI 192.168.202.120@tcp114 [15794.080224] LNet: Removed LNI 192.168.202.120@tcp115 [15795.104582] LNet: Removed LNI 192.168.202.120@tcp116 [15796.128244] LNet: Removed LNI 192.168.202.120@tcp117 [15797.152220] LNet: Removed LNI 192.168.202.120@tcp118 [15797.183334] LNetError: 468312:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15797.206940] LNetError: 468312:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15797.222581] LNet: Removed LNI 192.168.202.120@tcp119 [15798.241678] LNet: Removed LNI 192.168.202.120@tcp120 [15799.267287] LNet: Removed LNI 192.168.202.120@tcp121 [15800.288205] LNet: Removed LNI 192.168.202.120@tcp122 [15801.315567] LNet: Removed LNI 192.168.202.120@tcp123 [15802.336187] LNet: Removed LNI 192.168.202.120@tcp124 [15803.360289] LNet: Removed LNI 192.168.202.120@tcp125 [15804.386353] LNet: Removed LNI 192.168.202.120@tcp126 [15805.408770] LNet: Removed LNI 192.168.202.120@tcp127 [15806.432610] LNet: Removed LNI 192.168.202.120@tcp128 [15807.456593] LNet: Removed LNI 192.168.202.120@tcp129 [15807.460139] LNet: Removed LNI 192.168.202.120@tcp130 [15807.463291] LNet: Removed LNI 192.168.202.120@tcp131 [15807.473485] LNet: Removed LNI 192.168.202.120@tcp132 [15807.479393] LNet: Removed LNI 192.168.202.120@tcp133 [15807.487230] LNet: Removed LNI 192.168.202.120@tcp134 [15807.508256] LNet: Removed LNI 192.168.202.120@tcp135 [15808.900132] Key type .llcrypt unregistered [15808.903836] Key type ._llcrypt unregistered [15837.683373] Key type ._llcrypt registered [15837.686985] Key type .llcrypt registered [15837.889795] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [15854.835272] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [15855.689542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15855.717074] alg: No test for adler32 (adler32-zlib) [15857.270772] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [15857.928108] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [15859.824185] Key type lgssc registered [15861.179443] Lustre: Echo OBD driver; http://www.lustre.org/ [15906.706403] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 07:06:57 (1787396817) [15930.799879] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [15934.193393] Key type lgssc unregistered [15934.493252] LNet: 472418:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15934.520908] LNetError: 472418:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15934.569027] LNet: Removed LNI 192.168.202.120@tcp [15935.564747] Key type .llcrypt unregistered [15935.568391] Key type ._llcrypt unregistered [15956.635836] Key type ._llcrypt registered [15956.639849] Key type .llcrypt registered [15956.785622] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [15958.263768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15958.386329] alg: No test for adler32 (adler32-zlib) [15959.509323] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [15959.789637] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [15961.521347] Key type lgssc registered [15962.703461] Lustre: Echo OBD driver; http://www.lustre.org/ [15974.183873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15974.219910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15975.605709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15975.641639] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15975.751365] Lustre: lustre-MDT0000: new disk, initializing [15975.846967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15975.880325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15981.066390] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [15993.484037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15993.777229] Lustre: 473634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15993.927556] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15993.938454] Lustre: Skipped 1 previous similar message [15994.055361] Lustre: lustre-MDT0001: new disk, initializing [15994.305936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15994.400919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15994.414454] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16000.634854] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [16014.032778] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16021.095302] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16029.702851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16030.068342] Lustre: lustre-OST0000: new disk, initializing [16030.072631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16030.081258] Lustre: 475601:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16030.214175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16036.991699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16037.004436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16037.200235] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16038.205219] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [16049.828951] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16055.229308] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing lnet_if_list [16062.097900] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [16062.159475] LNet: Added LNI 192.168.202.120@tcp10 [8/256/0/180] [16069.340745] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [16069.372966] LNet: Added LNI 192.168.202.120@tcp11 [8/256/0/180] [16074.094709] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [16074.124544] LNet: Added LNI 192.168.202.120@tcp12 [8/256/0/180] [16080.330222] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [16080.356641] LNet: Added LNI 192.168.202.120@tcp13 [8/256/0/180] [16086.124513] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [16086.172599] LNet: Added LNI 192.168.202.120@tcp14 [8/256/0/180] [16091.966638] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [16091.998138] LNet: Added LNI 192.168.202.120@tcp15 [8/256/0/180] [16098.201716] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [16098.223663] LNet: Added LNI 192.168.202.120@tcp16 [8/256/0/180] [16104.462418] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [16104.490791] LNet: Added LNI 192.168.202.120@tcp17 [8/256/0/180] [16109.705200] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [16109.724815] LNet: Added LNI 192.168.202.120@tcp18 [8/256/0/180] [16115.475312] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [16115.508655] LNet: Added LNI 192.168.202.120@tcp19 [8/256/0/180] [16121.697408] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [16121.728757] LNet: Added LNI 192.168.202.120@tcp20 [8/256/0/180] [16126.777702] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [16126.817506] LNet: Added LNI 192.168.202.120@tcp21 [8/256/0/180] [16132.103420] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [16132.169774] LNet: Added LNI 192.168.202.120@tcp22 [8/256/0/180] [16138.283430] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [16138.310154] LNet: Added LNI 192.168.202.120@tcp23 [8/256/0/180] [16145.884074] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [16145.980352] LNet: Added LNI 192.168.202.120@tcp24 [8/256/0/180] [16155.275144] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [16155.300682] LNet: Added LNI 192.168.202.120@tcp25 [8/256/0/180] [16161.865902] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [16161.901390] LNet: Added LNI 192.168.202.120@tcp26 [8/256/0/180] [16169.118774] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [16169.149454] LNet: Added LNI 192.168.202.120@tcp27 [8/256/0/180] [16175.509779] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [16175.537650] LNet: Added LNI 192.168.202.120@tcp28 [8/256/0/180] [16181.252141] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [16181.279686] LNet: Added LNI 192.168.202.120@tcp29 [8/256/0/180] [16187.449967] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [16187.495771] LNet: Added LNI 192.168.202.120@tcp30 [8/256/0/180] [16194.371260] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [16194.398173] LNet: Added LNI 192.168.202.120@tcp31 [8/256/0/180] [16201.640904] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [16201.706609] LNet: Added LNI 192.168.202.120@tcp32 [8/256/0/180] [16209.677757] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [16209.708213] LNet: Added LNI 192.168.202.120@tcp33 [8/256/0/180] [16217.090856] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [16217.158752] LNet: Added LNI 192.168.202.120@tcp34 [8/256/0/180] [16226.708612] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [16226.729354] LNet: Added LNI 192.168.202.120@tcp35 [8/256/0/180] [16237.886741] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16237.911481] LNet: Added LNI 192.168.202.120@tcp36 [8/256/0/180] [16247.229100] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16247.286279] LNet: Added LNI 192.168.202.120@tcp37 [8/256/0/180] [16256.908650] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16256.946862] LNet: Added LNI 192.168.202.120@tcp38 [8/256/0/180] [16266.105784] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16266.164979] LNet: Added LNI 192.168.202.120@tcp39 [8/256/0/180] [16277.706789] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16277.775774] LNet: Added LNI 192.168.202.120@tcp40 [8/256/0/180] [16288.107940] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16288.135148] LNet: Added LNI 192.168.202.120@tcp41 [8/256/0/180] [16297.310082] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16297.357372] LNet: Added LNI 192.168.202.120@tcp42 [8/256/0/180] [16305.416858] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16305.462708] LNet: Added LNI 192.168.202.120@tcp43 [8/256/0/180] [16314.866232] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16314.902501] LNet: Added LNI 192.168.202.120@tcp44 [8/256/0/180] [16323.019980] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16323.080748] LNet: Added LNI 192.168.202.120@tcp45 [8/256/0/180] [16333.995993] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16334.049574] LNet: Added LNI 192.168.202.120@tcp46 [8/256/0/180] [16345.876837] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16345.900158] LNet: Added LNI 192.168.202.120@tcp47 [8/256/0/180] [16357.668632] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16357.690105] LNet: Added LNI 192.168.202.120@tcp48 [8/256/0/180] [16367.457878] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16367.516811] LNet: Added LNI 192.168.202.120@tcp49 [8/256/0/180] [16377.147914] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16377.170657] LNet: Added LNI 192.168.202.120@tcp50 [8/256/0/180] [16398.825338] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16398.832519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16398.834509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16398.847296] Lustre: Skipped 1 previous similar message [16403.943950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16403.953622] Lustre: Skipped 4 previous similar messages [16405.149646] Lustre: server umount lustre-MDT0000 complete [16409.855812] LustreError: 473626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397322 with bad export cookie 13348601265939356386 [16409.858890] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16409.870352] LustreError: 473626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16410.264530] Lustre: server umount lustre-MDT0001 complete [16430.561233] Lustre: 472906:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787397326/real 1787397326] req@00000000bab2b830 x1874221462647168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787397342 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16430.600239] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16430.617319] Lustre: Skipped 1 previous similar message [16430.762856] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [16435.680139] Lustre: 472904:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787397331/real 1787397331] req@0000000078edf703 x1874221462647296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787397347 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16436.769477] Lustre: 472905:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787397332/real 1787397332] req@000000002c41581a x1874221462647552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787397348 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16440.800169] Lustre: 472907:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787397337/real 1787397337] req@00000000d2459e55 x1874221462647680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787397353 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16441.487509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16445.984735] Lustre: 472907:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787397342/real 1787397342] req@000000000b485e01 x1874221462647808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787397358 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16455.212729] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb93fc220a0a52500 [16455.226303] Lustre: MGC192.168.202.120@tcp: Connection restored to 0@lo (at 0@lo) [16455.564025] LustreError: 475595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16455.594494] LustreError: 475595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [16455.641248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16460.768655] LustreError: 475596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16461.310688] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [16463.848220] LustreError: 475595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16463.852987] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16468.962514] LustreError: 475596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16471.023874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16471.497330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16475.634089] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16476.438555] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [16485.274426] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16491.312510] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16516.580102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16516.609571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16520.483391] Lustre: server umount lustre-OST0000 complete [16526.820878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16526.835065] Lustre: Skipped 1 previous similar message [16526.841673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16526.852037] Lustre: Skipped 2 previous similar messages [16531.138903] Lustre: server umount lustre-MDT0000 complete [16531.938442] LustreError: 499859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16531.961960] LustreError: 499859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [16535.738632] LustreError: 474428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397447 with bad export cookie 13348601265939358976 [16535.743075] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16535.747272] LustreError: 474428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16536.293422] Lustre: server umount lustre-MDT0001 complete [16550.292514] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [16552.938612] Key type lgssc unregistered [16553.477179] LNet: 502658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16553.501861] LNetError: 502658:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16554.528880] LNet: Removed LNI 192.168.202.120@tcp [16554.530761] LNetError: 502658:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16554.545173] LNet: Removed LNI 192.168.202.120@tcp10 [16554.555198] LNet: Removed LNI 192.168.202.120@tcp11 [16554.561435] LNet: Removed LNI 192.168.202.120@tcp12 [16554.567504] LNet: Removed LNI 192.168.202.120@tcp13 [16554.571394] LNet: Removed LNI 192.168.202.120@tcp14 [16554.574334] LNet: Removed LNI 192.168.202.120@tcp15 [16554.580048] LNet: Removed LNI 192.168.202.120@tcp16 [16554.583564] LNet: Removed LNI 192.168.202.120@tcp17 [16554.588906] LNet: Removed LNI 192.168.202.120@tcp18 [16554.592601] LNet: Removed LNI 192.168.202.120@tcp19 [16554.603623] LNet: Removed LNI 192.168.202.120@tcp20 [16554.608567] LNet: Removed LNI 192.168.202.120@tcp21 [16554.612319] LNet: Removed LNI 192.168.202.120@tcp22 [16554.617442] LNet: Removed LNI 192.168.202.120@tcp23 [16554.621510] LNet: Removed LNI 192.168.202.120@tcp24 [16554.626214] LNet: Removed LNI 192.168.202.120@tcp25 [16554.635386] LNet: Removed LNI 192.168.202.120@tcp26 [16554.647042] LNet: Removed LNI 192.168.202.120@tcp27 [16554.651252] LNet: Removed LNI 192.168.202.120@tcp28 [16554.661614] LNet: Removed LNI 192.168.202.120@tcp29 [16554.668867] LNet: Removed LNI 192.168.202.120@tcp30 [16554.674381] LNet: Removed LNI 192.168.202.120@tcp31 [16554.679194] LNet: Removed LNI 192.168.202.120@tcp32 [16554.694827] LNet: Removed LNI 192.168.202.120@tcp33 [16554.705163] LNet: Removed LNI 192.168.202.120@tcp34 [16554.734631] LNet: Removed LNI 192.168.202.120@tcp35 [16554.746995] LNet: Removed LNI 192.168.202.120@tcp36 [16554.750348] LNet: Removed LNI 192.168.202.120@tcp37 [16554.755989] LNet: Removed LNI 192.168.202.120@tcp38 [16554.761343] LNet: Removed LNI 192.168.202.120@tcp39 [16554.765591] LNet: Removed LNI 192.168.202.120@tcp40 [16554.769147] LNet: Removed LNI 192.168.202.120@tcp41 [16554.771990] LNet: Removed LNI 192.168.202.120@tcp42 [16554.782773] LNet: Removed LNI 192.168.202.120@tcp43 [16554.784717] LNet: Removed LNI 192.168.202.120@tcp44 [16554.796880] LNet: Removed LNI 192.168.202.120@tcp45 [16554.800672] LNet: Removed LNI 192.168.202.120@tcp46 [16554.804129] LNet: Removed LNI 192.168.202.120@tcp47 [16554.807856] LNet: Removed LNI 192.168.202.120@tcp48 [16554.815515] LNet: Removed LNI 192.168.202.120@tcp49 [16554.831667] LNet: Removed LNI 192.168.202.120@tcp50 [16555.784942] Key type .llcrypt unregistered [16555.790547] Key type ._llcrypt unregistered [16581.224828] Key type ._llcrypt registered [16581.231293] Key type .llcrypt registered [16581.412719] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [16601.065760] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [16602.506655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16602.613282] alg: No test for adler32 (adler32-zlib) [16603.757634] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [16604.145125] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [16605.889117] Key type lgssc registered [16607.070127] Lustre: Echo OBD driver; http://www.lustre.org/ [16648.002556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16648.023837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16649.356620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16649.393405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16649.532796] Lustre: lustre-MDT0000: new disk, initializing [16649.706538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16649.737832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16655.106625] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [16666.557701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16666.704296] Lustre: 505858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16666.858396] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16666.865696] Lustre: Skipped 1 previous similar message [16666.937603] Lustre: lustre-MDT0001: new disk, initializing [16667.011257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16667.037745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16667.050554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16670.659981] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [16682.855768] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16693.513472] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16705.866682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16706.099748] Lustre: *** cfs_fail_loc=90f, val=0*** [16706.186921] Lustre: lustre-OST0000: new disk, initializing [16706.190134] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16706.200579] Lustre: 507874:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16706.317056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16712.309355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16712.340810] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16712.499392] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16714.876387] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [16727.863813] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16738.282051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16738.288851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16738.340379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16738.360751] Lustre: Skipped 1 previous similar message [16742.674661] Lustre: server umount lustre-OST0000 complete [16748.516135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16748.527512] Lustre: Skipped 1 previous similar message [16748.535364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16748.550124] Lustre: Skipped 1 previous similar message [16753.637821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16753.649106] Lustre: Skipped 1 previous similar message [16754.504559] Lustre: server umount lustre-MDT0000 complete [16758.884691] LustreError: 505848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397671 with bad export cookie 18305432370337087135 [16758.899696] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16759.371703] Lustre: server umount lustre-MDT0001 complete [16773.876338] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [16777.458361] Key type lgssc unregistered [16777.749290] LNet: 509781:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16777.757424] LNetError: 509781:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16777.779375] LNet: Removed LNI 192.168.202.120@tcp [16778.608528] Key type .llcrypt unregistered [16778.616231] Key type ._llcrypt unregistered [16804.462481] Key type ._llcrypt registered [16804.465995] Key type .llcrypt registered [16804.677251] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [16807.426397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16807.610612] alg: No test for adler32 (adler32-zlib) [16809.181430] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [16809.657731] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [16811.376414] Key type lgssc registered [16813.099904] Lustre: Echo OBD driver; http://www.lustre.org/ [16823.718930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16823.735884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16825.169452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16829.973763] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [16835.563900] LustreError: 511053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16835.600698] LustreError: 511053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [16840.683684] LustreError: 511052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16841.223709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16841.672552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16847.403594] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [16857.063522] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16864.774585] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16874.608882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16875.234806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16884.508528] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [16894.103315] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16907.146670] loop0: detected capacity change from 0 to 204800000 [16907.415482] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16907.426829] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16907.447128] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16915.175598] loop0: detected capacity change from 0 to 204800000 [16915.470016] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16915.480952] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16915.494675] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16922.306478] loop0: detected capacity change from 0 to 204800000 [16928.718304] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16928.960821] Lustre: fs15246-OST0000: new disk, initializing [16928.967971] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16928.978967] Lustre: 514651:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16929.069870] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16936.810722] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [16949.308432] loop1: detected capacity change from 0 to 204800000 [16956.659172] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16956.870855] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16956.949012] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16957.089933] Lustre: fs15246-MDT0000: new disk, initializing [16957.374774] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16957.405220] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16957.572867] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16958.627921] Lustre: 515775:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000009817c754 x1874222352749056/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787397886 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16962.985695] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [16967.786282] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16967.804210] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16967.827409] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16989.688124] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16989.703088] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16994.799355] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16995.204252] Lustre: server umount fs15246-MDT0000 complete [16999.909492] LustreError: 511048:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17005.039141] LustreError: 511048:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17007.391658] Lustre: server umount fs15246-OST0000 complete [17023.969583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17023.987689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17024.015765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17024.997942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17025.000834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17027.266167] Lustre: server umount lustre-OST0000 complete [17035.747359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17035.750066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17035.780375] Lustre: Skipped 1 previous similar message [17035.804783] Lustre: Skipped 4 previous similar messages [17038.706831] Lustre: server umount lustre-MDT0000 complete [17040.866543] LustreError: 515658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17040.898139] LustreError: 515658:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [17043.211909] LustreError: 515769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397955 with bad export cookie 2919946508559956860 [17043.214375] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17043.222170] LustreError: 515769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17043.692251] Lustre: server umount lustre-MDT0001 complete [17058.226800] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [17062.228709] Key type lgssc unregistered [17062.577724] LNet: 518873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17062.584484] LNetError: 518873:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [17062.600540] LNet: Removed LNI 192.168.202.120@tcp [17063.848883] Key type .llcrypt unregistered [17063.854223] Key type ._llcrypt unregistered [17086.072327] Key type ._llcrypt registered [17086.079232] Key type .llcrypt registered [17086.236490] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [17087.577607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17087.657781] alg: No test for adler32 (adler32-zlib) [17088.924447] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [17089.184260] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [17090.872258] Key type lgssc registered [17092.367601] Lustre: Echo OBD driver; http://www.lustre.org/ [17143.379900] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [17155.419751] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [17200.657775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17200.682143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17201.927966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17201.966501] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17202.083136] Lustre: lustre-MDT0000: new disk, initializing [17202.272431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17202.324317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17207.925084] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [17221.156945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17221.350104] Lustre: 523345:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17221.474151] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17221.488029] Lustre: Skipped 1 previous similar message [17221.620789] Lustre: lustre-MDT0001: new disk, initializing [17221.795169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17221.848363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17221.862414] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17227.362668] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [17238.570363] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [17245.623204] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [17254.793646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17255.350218] Lustre: lustre-OST0000: new disk, initializing [17255.357630] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17255.381617] Lustre: 525314:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17255.562174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17262.264559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17262.279123] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17262.472766] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17264.478216] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [17274.718798] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [17280.185932] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [17280.359658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17285.431273] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17285.675609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17288.178693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17288.185587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17288.191511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17288.196670] Lustre: Skipped 1 previous similar message [17288.217286] LustreError: Skipped 1 previous similar message [17293.293798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17293.301646] Lustre: Skipped 1 previous similar message [17294.385160] Lustre: server umount lustre-OST0000 complete [17303.523065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17303.527223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17303.575121] Lustre: Skipped 1 previous similar message [17303.578164] Lustre: Skipped 1 previous similar message [17303.608756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17305.407437] Lustre: server umount lustre-MDT0000 complete [17309.977333] LustreError: 524141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398222 with bad export cookie 6307857054143712264 [17309.984535] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17309.988514] LustreError: 524141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17310.233943] Lustre: server umount lustre-MDT0001 complete [17341.303280] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [17353.087324] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [17390.545253] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 17079 sec ========= 07:31:40 (1787398300) [17392.279965] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:31:43 (1787398303) === [17396.521707] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:31:47 (1787398307) === [17443.752448] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [17447.455436] Key type lgssc unregistered [17447.783338] LNet: 531960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17447.802077] LNetError: 531960:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [17447.822456] LNet: Removed LNI 192.168.202.120@tcp [17449.172882] Key type .llcrypt unregistered [17449.176434] Key type ._llcrypt unregistered