[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 494157076 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003195] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010136] LSM: Security Framework initializing [ 0.011058] Yama: becoming mindful. [ 0.012041] SELinux: Initializing. [ 0.013075] *** VALIDATE selinux *** [ 0.022302] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027889] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028184] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029121] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031067] *** VALIDATE tmpfs *** [ 0.032443] *** VALIDATE proc *** [ 0.033243] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.036277] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039033] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043352] debug: unmapping init [mem 0xffffffff9e459000-0xffffffff9e460fff] [ 0.046222] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047766] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048025] ... version: 2 [ 0.049011] ... bit width: 48 [ 0.049913] ... generic registers: 4 [ 0.050010] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052014] ... fixed-purpose events: 3 [ 0.053015] ... event mask: 000000070000000f [ 0.054304] rcu: Hierarchical SRCU implementation. [ 0.056384] smp: Bringing up secondary CPUs ... [ 0.057557] x86: Booting SMP configuration: [ 0.058022] .... node #0, CPUs: #1 #2 #3 [ 0.061293] smp: Brought up 1 node, 4 CPUs [ 0.063015] smpboot: Max logical packages: 1 [ 0.064016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145021] node 0 deferred pages initialised in 79ms [ 0.148103] devtmpfs: initialized [ 0.150330] x86/mm: Memory block size: 128MB [ 0.155176] gcov: version magic: 0x41383552 [ 0.158310] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.163110] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.167404] pinctrl core: initialized pinctrl subsystem [ 0.170213] [ 0.171010] ************************************************************* [ 0.175015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.179014] ** ** [ 0.183015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.186013] ** ** [ 0.190015] ** This means that this kernel is built to expose internal ** [ 0.193013] ** IOMMU data structures, which may compromise security on ** [ 0.197022] ** your system. ** [ 0.200013] ** ** [ 0.202011] ** If you see this message and you are not debugging the ** [ 0.205013] ** kernel, report this immediately to your vendor! ** [ 0.207012] ** ** [ 0.210023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.213015] ************************************************************* [ 0.217451] NET: Registered protocol family 16 [ 0.219513] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.221064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.223038] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.225097] cpuidle: using governor menu [ 0.226658] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.229431] PCI: Using configuration type 1 for base access [ 0.231100] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.239248] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.240017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.242079] cryptd: max_cpu_qlen set to 1000 [ 0.248262] ACPI: Added _OSI(Module Device) [ 0.249011] ACPI: Added _OSI(Processor Device) [ 0.251009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.252008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.255430] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.261353] ACPI: Interpreter enabled [ 0.263067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.264012] ACPI: Using IOAPIC for interrupt routing [ 0.266104] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.270368] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.278137] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.281049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.286022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.291080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.296426] acpiphp: Slot [2] registered [ 0.298130] acpiphp: Slot [5] registered [ 0.299129] acpiphp: Slot [6] registered [ 0.300163] acpiphp: Slot [7] registered [ 0.302122] acpiphp: Slot [8] registered [ 0.304155] acpiphp: Slot [9] registered [ 0.306152] acpiphp: Slot [10] registered [ 0.310137] acpiphp: Slot [3] registered [ 0.311135] acpiphp: Slot [4] registered [ 0.314120] acpiphp: Slot [11] registered [ 0.315128] acpiphp: Slot [12] registered [ 0.317113] acpiphp: Slot [13] registered [ 0.318106] acpiphp: Slot [14] registered [ 0.319078] acpiphp: Slot [15] registered [ 0.321118] acpiphp: Slot [16] registered [ 0.322116] acpiphp: Slot [17] registered [ 0.324128] acpiphp: Slot [18] registered [ 0.325090] acpiphp: Slot [19] registered [ 0.327117] acpiphp: Slot [20] registered [ 0.328101] acpiphp: Slot [21] registered [ 0.330122] acpiphp: Slot [22] registered [ 0.332132] acpiphp: Slot [23] registered [ 0.333117] acpiphp: Slot [24] registered [ 0.335156] acpiphp: Slot [25] registered [ 0.337126] acpiphp: Slot [26] registered [ 0.338123] acpiphp: Slot [27] registered [ 0.340088] acpiphp: Slot [28] registered [ 0.342145] acpiphp: Slot [29] registered [ 0.344294] acpiphp: Slot [30] registered [ 0.346242] acpiphp: Slot [31] registered [ 0.348070] PCI host bridge to bus 0000:00 [ 0.349019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.351017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.353025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.356024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.359024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.360020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.362155] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.365043] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.368250] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.378014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.382499] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.385018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.388020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.391017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.393645] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.396785] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.399029] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.401938] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.409017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.421015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.426013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.432136] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.438018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.445017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.462012] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.474143] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.483013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.489014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.504016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.512602] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.519013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.524015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.543015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.552520] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.559016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.566016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.586016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.600138] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.609020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.616016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.650016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.660304] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.668015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.673027] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.691020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.702098] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.704420] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.707355] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.710378] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.713270] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.718135] iommu: Default domain type: Passthrough [ 0.721483] SCSI subsystem initialized [ 0.723129] ACPI: bus type USB registered [ 0.724118] usbcore: registered new interface driver usbfs [ 0.726076] usbcore: registered new interface driver hub [ 0.728071] usbcore: registered new device driver usb [ 0.730202] pps_core: LinuxPPS API ver. 1 registered [ 0.732012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.735065] PTP clock support registered [ 0.737126] EDAC MC: Ver: 3.0.0 [ 0.739452] PCI: Using ACPI for IRQ routing [ 0.740325] NetLabel: Initializing [ 0.741000] NetLabel: domain hash size = 128 [ 0.743016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.745073] NetLabel: unlabeled traffic allowed by default [ 0.747129] vgaarb: loaded [ 0.749280] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.751013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.757312] clocksource: Switched to clocksource kvm-clock [ 0.872515] VFS: Disk quotas dquot_6.6.0 [ 0.874086] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.876665] *** VALIDATE ramfs *** [ 0.877913] *** VALIDATE hugetlbfs *** [ 0.879952] pnp: PnP ACPI init [ 0.882465] pnp: PnP ACPI: found 6 devices [ 0.908896] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.912292] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.914485] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.916783] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.919157] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.921634] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.924413] NET: Registered protocol family 2 [ 0.926948] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.931716] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.935173] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.940771] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.944184] TCP: Hash tables configured (established 65536 bind 65536) [ 0.947334] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.950856] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.953692] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.956679] NET: Registered protocol family 1 [ 0.960907] RPC: Registered named UNIX socket transport module. [ 0.963269] RPC: Registered udp transport module. [ 0.965187] RPC: Registered tcp transport module. [ 0.966956] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.969642] NET: Registered protocol family 44 [ 0.971230] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.973452] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.975547] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.978277] PCI: CLS 0 bytes, default 64 [ 0.980164] Unpacking initramfs... [ 2.537151] debug: unmapping init [mem 0xffff98083cc54000-0xffff98083ffbffff] [ 2.541236] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.543494] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.546568] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.039319] Initialise system trusted keyrings [ 3.042425] Key type blacklist registered [ 3.044440] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.059478] zbud: loaded [ 3.064496] *** VALIDATE nfs *** [ 3.065906] *** VALIDATE nfs4 *** [ 3.067721] pstore: using deflate compression [ 3.074078] Platform Keyring initialized [ 3.182620] NET: Registered protocol family 38 [ 3.184968] Key type asymmetric registered [ 3.186207] Asymmetric key parser 'x509' registered [ 3.187634] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.190094] io scheduler mq-deadline registered [ 3.191963] io scheduler kyber registered [ 3.193662] io scheduler bfq registered [ 3.195574] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.199266] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.202622] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.206128] ACPI: Power Button [PWRF] [ 3.211791] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.218273] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.234635] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.242624] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.261630] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.289177] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.316784] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.322291] Non-volatile memory driver v1.3 [ 3.324252] Linux agpgart interface v0.103 [ 3.356485] virtio_blk virtio1: [vda] 149968 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.359704] vda: detected capacity change from 0 to 76783616 [ 3.376216] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.379348] vdb: detected capacity change from 0 to 1073741824 [ 3.392903] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.395975] vdc: detected capacity change from 0 to 2621440000 [ 3.412881] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.415578] vdd: detected capacity change from 0 to 2621440000 [ 3.429369] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.431903] vde: detected capacity change from 0 to 4294967296 [ 3.445822] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.449351] vdf: detected capacity change from 0 to 4294967296 [ 3.455206] libphy: Fixed MDIO Bus: probed [ 3.459813] usbcore: registered new interface driver usbserial_generic [ 3.461941] usbserial: USB Serial support registered for generic [ 3.463891] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.467981] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.469534] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.471926] mousedev: PS/2 mouse device common for all mice [ 3.475174] rtc_cmos 00:05: RTC can wake from S4 [ 3.478157] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.483055] rtc_cmos 00:05: registered as rtc0 [ 3.485265] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.485601] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.490935] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.491779] intel_pstate: CPU model not supported [ 3.498850] hid: raw HID events driver (C) Jiri Kosina [ 3.500664] usbcore: registered new interface driver usbhid [ 3.502364] usbhid: USB HID core driver [ 3.503630] drop_monitor: Initializing network drop monitor service [ 3.505938] Initializing XFRM netlink socket [ 3.508219] NET: Registered protocol family 10 [ 3.511463] Segment Routing with IPv6 [ 3.513091] NET: Registered protocol family 17 [ 3.514851] mpls_gso: MPLS GSO support [ 3.520396] RAS: Correctable Errors collector initialized. [ 3.522693] AVX version of gcm_enc/dec engaged. [ 3.524305] AES CTR mode by8 optimization enabled [ 3.598921] sched_clock: Marking stable (3598844701, 0)->(4534595850, -935751149) [ 3.603175] registered taskstats version 1 [ 3.606271] Loading compiled-in X.509 certificates [ 3.608455] zswap: loaded using pool lzo/zbud [ 3.632762] Key type big_key registered [ 3.647853] Key type encrypted registered [ 3.649213] ima: No TPM chip found, activating TPM-bypass! [ 3.651239] ima: Allocated hash algorithm: sha1 [ 3.652778] ima: No architecture policies found [ 3.654513] evm: Initialising EVM extended attributes: [ 3.656471] evm: security.selinux [ 3.657807] evm: security.ima [ 3.658653] evm: security.capability [ 3.659704] evm: HMAC attrs: 0x1 [ 3.662287] rtc_cmos 00:05: setting system clock to 2026-09-10 18:44:13 UTC (1789065853) [ 3.667286] debug: unmapping init [mem 0xffffffff9f403000-0xffffffff9f5fffff] [ 3.670377] debug: unmapping init [mem 0xffffffff9e182000-0xffffffff9e458fff] [ 3.677155] Write protecting the kernel read-only data: 28672k [ 3.680372] debug: unmapping init [mem 0xffffffff9c803000-0xffffffff9c9fffff] [ 3.683140] debug: unmapping init [mem 0xffffffff9d114000-0xffffffff9d1fffff] [ 3.717103] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.725419] systemd[1]: Detected virtualization kvm. [ 3.726859] systemd[1]: Detected architecture x86-64. [ 3.728756] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.755104] systemd[1]: No hostname configured. [ 3.757310] systemd[1]: Set hostname to . [ 3.759566] random: systemd: uninitialized urandom read (16 bytes read) [ 3.762362] systemd[1]: Initializing machine ID from random generator. [ 3.807888] random: ln: uninitialized urandom read (6 bytes read) [ 3.914945] random: systemd: uninitialized urandom read (16 bytes read) [ 3.918546] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.923563] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.927489] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.609412] device-mapper: uevent: version 1.0.3 [ 4.611963] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.353317] random: fast init done [ 5.380451] virtio_net virtio0 ens2: renamed from eth0 [ 5.391900] scsi host0: ata_piix [ 5.399135] scsi host1: ata_piix [ 5.400831] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.403060] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.961399] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.167950] random: crng init done [ 10.169132] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.760135] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.039247] printk: systemd: 24 output lines suppressed due to ratelimiting [ 12.310722] SELinux: Disabled at runtime. [ 12.379370] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.388742] systemd[1]: Detected virtualization kvm. [ 12.391084] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.938001] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.941962] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.947885] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.952713] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.956871] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.966345] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.975519] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server A[ 13.038995] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ctivation Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.425140] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.751879] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.822341] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.925725] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.945458] EDAC sbridge: Ver: 1.1.2 [ 15.738780] Key type dns_resolver registered [ 16.053544] NFS: Registering the id_resolver key type [ 16.055686] Key type id_resolver registered [ 16.057272] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg337-server login: [ 42.287293] libcfs: loading out-of-tree module taints kernel. [ 42.308797] Key type ._llcrypt registered [ 42.310436] Key type .llcrypt registered [ 42.360855] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 51.260712] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 51.921264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.929996] alg: No test for adler32 (adler32-zlib) [ 52.994132] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 53.380310] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 55.023237] Key type lgssc registered [ 55.776175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 75.470694] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 89.191022] hrtimer: interrupt took 5520509 ns [ 121.423249] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 134.114208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 134.167749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 135.544967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 135.614991] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 135.776758] Lustre: lustre-MDT0000: new disk, initializing [ 135.902492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 135.957181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 140.245693] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 153.485984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.628104] Lustre: 6478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 153.670489] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 153.676980] Lustre: Skipped 1 previous similar message [ 153.840776] Lustre: lustre-MDT0001: new disk, initializing [ 153.976637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 154.005943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 154.026816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 158.486461] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 163.165482] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 172.301147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 172.654559] Lustre: lustre-OST0000: new disk, initializing [ 172.658988] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 172.666643] Lustre: 8418:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 172.739743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.708193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 177.731299] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 177.806205] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 178.123960] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.775337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 191.895774] Lustre: lustre-OST0001: new disk, initializing [ 191.898796] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 191.902723] Lustre: 9487:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 191.955295] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 197.928985] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.699105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 199.709389] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 199.823691] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 210.116933] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 219.871119] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 227.281625] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing check_logdir /tmp/testlogs/ [ 233.799091] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing yml_node [ 240.648568] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 243.734279] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 246.315458] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 247.832876] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 10 14:48:16 EDT 2026 [ 265.358933] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 266.738689] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 286.689031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.691577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 286.717029] Lustre: Skipped 1 previous similar message [ 286.735437] Lustre: Skipped 3 previous similar messages [ 289.028981] Lustre: server umount lustre-MDT0000 complete [ 292.558759] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066142 with bad export cookie 12139555249811776404 [ 292.564192] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 292.576575] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 296.931188] LustreError: 6486:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 296.931371] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.940073] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 296.952844] LustreError: 6486:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 296.976837] Lustre: Skipped 3 previous similar messages [ 297.952956] LustreError: 6485:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.953029] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 297.982456] Lustre: Skipped 2 previous similar messages [ 298.939310] Lustre: server umount lustre-MDT0001 complete [ 302.626615] Lustre: server umount lustre-OST0000 complete [ 306.240972] Lustre: server umount lustre-OST0001 complete [ 312.321306] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 322.287818] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 356.984591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 357.266439] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 357.297739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 357.369496] Lustre: lustre-MDT0000: new disk, initializing [ 357.425441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 357.439499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 362.954484] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 373.932035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 374.073229] Lustre: 15042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 374.141966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 374.148808] Lustre: Skipped 1 previous similar message [ 374.270401] Lustre: lustre-MDT0001: new disk, initializing [ 374.400114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 374.446655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 374.469974] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 378.938877] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 388.408741] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 394.704705] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 402.872614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 403.283324] Lustre: lustre-OST0000: new disk, initializing [ 403.297290] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 403.321677] Lustre: 17015:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 403.499542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 404.816598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 404.854557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 405.185639] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 410.549297] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 420.697397] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 425.081498] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 425.371767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 429.908824] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 430.200375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 435.694073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 435.696247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 435.705810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 435.729592] Lustre: Skipped 1 previous similar message [ 435.760700] LustreError: Skipped 1 previous similar message [ 438.555378] Lustre: server umount lustre-OST0000 complete [ 445.928816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 445.942041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 445.944700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 445.944709] Lustre: Skipped 1 previous similar message [ 445.961716] Lustre: Skipped 1 previous similar message [ 448.380610] Lustre: server umount lustre-MDT0000 complete [ 451.045234] LustreError: 15048:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 451.067304] LustreError: 15048:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 451.998357] LustreError: 15840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066301 with bad export cookie 12139555249811777937 [ 452.005560] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 452.009481] LustreError: 15840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 452.310745] Lustre: server umount lustre-MDT0001 complete [ 456.476656] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 14:51:44 (1789066304) [ 459.111208] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 460.762441] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 14:51:49 (1789066309) [ 467.194201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 467.656162] LustreError: 19427:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 467.728299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 472.163027] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 473.068032] LustreError: 19428:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 481.066125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 486.101914] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 493.925967] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 499.539532] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 507.763726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 515.280852] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 524.647494] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 537.069201] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 537.086853] Lustre: Skipped 1 previous similar message [ 537.097532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 537.107039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 537.115555] Lustre: Skipped 1 previous similar message [ 541.836514] Lustre: server umount lustre-OST0000 complete [ 547.298298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 547.346520] Lustre: Skipped 1 previous similar message [ 551.714929] Lustre: server umount lustre-MDT0000 complete [ 552.426588] LustreError: 20155:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 552.470416] LustreError: 20155:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 556.439674] LustreError: 19407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066406 with bad export cookie 12139555249811778819 [ 556.444631] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 556.453618] LustreError: 19407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 556.901683] Lustre: server umount lustre-MDT0001 complete [ 570.033582] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 573.842953] Key type lgssc unregistered [ 574.324701] LNet: 22961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 574.334493] LNetError: 22961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 574.356472] LNet: Removed LNI 192.168.203.137@tcp [ 575.580178] Key type .llcrypt unregistered [ 575.586149] Key type ._llcrypt unregistered [ 600.634898] Key type ._llcrypt registered [ 600.636505] Key type .llcrypt registered [ 600.740744] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 601.990661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 602.011416] alg: No test for adler32 (adler32-zlib) [ 603.129149] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 603.471397] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 605.199570] Key type lgssc registered [ 606.475393] Lustre: Echo OBD driver; http://www.lustre.org/ [ 615.300530] loop: module loaded [ 615.390388] loop0: detected capacity change from 0 to 204800000 [ 615.607993] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 615.626354] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 615.988864] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 622.711519] loop0: detected capacity change from 0 to 204800000 [ 622.874352] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 622.898639] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 622.916283] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 628.281460] loop0: detected capacity change from 0 to 204800000 [ 635.080095] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 635.127384] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 636.475354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 636.541619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 636.687585] Lustre: lustre-MDT0000: new disk, initializing [ 636.783520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 636.808973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 641.768399] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 650.882747] loop1: detected capacity change from 0 to 204800000 [ 656.571611] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 656.809574] Lustre: lustre-OST07c6: new disk, initializing [ 656.813116] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 656.817186] Lustre: Skipped 1 previous similar message [ 656.821280] Lustre: 26143:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 656.897653] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 665.310967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 665.334197] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 665.386662] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 665.399290] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 670.291859] Lustre: 25131:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 670.305425] Lustre: 25131:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 670.310359] Lustre: 25131:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 670.316798] Lustre: 25131:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 670.330037] Lustre: 25131:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 670.345463] Lustre: 25131:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 693.552430] Lustre: 25131:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 693.558852] Lustre: 25131:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 693.563119] Lustre: 25131:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 693.568516] Lustre: 25131:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 693.573921] Lustre: 25131:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 693.578879] Lustre: 25131:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 698.915938] Lustre: 25130:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 698.932585] Lustre: 25130:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 698.944614] Lustre: 25130:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 698.956796] Lustre: 25130:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.964459] Lustre: 25130:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 698.987553] Lustre: 25130:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.000251] Lustre: 25130:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 699.006614] Lustre: 25130:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.014464] Lustre: 25130:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 699.021406] Lustre: 25130:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.033631] Lustre: 25130:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 699.042449] Lustre: 25130:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.241967] Lustre: 25129:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 702.247625] Lustre: 25129:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 702.254235] Lustre: 25129:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 702.266278] Lustre: 25129:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.276080] Lustre: 25129:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 702.285138] Lustre: 25129:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.296792] Lustre: 25129:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 702.302899] Lustre: 25129:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.310074] Lustre: 25129:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 702.316951] Lustre: 25129:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.324453] Lustre: 25129:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 702.331293] Lustre: 25129:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 710.222936] Lustre: 25129:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 710.242857] Lustre: 25129:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 710.247924] Lustre: 25129:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 710.263649] Lustre: 25129:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 710.276707] Lustre: 25129:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 710.289893] Lustre: 25129:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 710.297344] Lustre: 25129:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 710.307477] Lustre: 25129:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 710.312666] Lustre: 25129:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 710.317198] Lustre: 25129:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 710.321884] Lustre: 25129:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 710.326225] Lustre: 25129:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 718.326683] Lustre: 25129:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 718.340429] Lustre: 25129:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 718.349192] Lustre: 25129:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 718.362952] Lustre: 25129:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 718.373586] Lustre: 25129:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 718.380172] Lustre: 25129:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 718.392149] Lustre: 25129:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 718.402104] Lustre: 25129:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 718.408769] Lustre: 25129:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 718.414340] Lustre: 25129:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 718.420607] Lustre: 25129:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 718.427090] Lustre: 25129:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 767.461430] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 767.491164] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 771.712651] Lustre: server umount lustre-OST07c6 complete [ 782.440521] Lustre: server umount lustre-MDT0000 complete [ 800.880410] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 804.295360] Key type lgssc unregistered [ 804.486402] LNet: 28704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.493777] LNetError: 28704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 804.510000] LNet: Removed LNI 192.168.203.137@tcp [ 805.412200] Key type .llcrypt unregistered [ 805.418493] Key type ._llcrypt unregistered [ 825.840879] Key type ._llcrypt registered [ 825.842909] Key type .llcrypt registered [ 826.037428] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 827.072054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 827.081217] alg: No test for adler32 (adler32-zlib) [ 828.170377] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 828.450139] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 830.111270] Key type lgssc registered [ 830.965722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 837.000466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 837.020787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 838.384638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 842.786191] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 848.871257] LustreError: 29971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 848.890210] LustreError: 29971:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 851.743959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 852.310474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 857.406147] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 866.623124] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 873.022869] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 880.416410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 880.975711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 884.108505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 887.927131] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 895.888653] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 912.866707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 912.877535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 912.903225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 912.911910] Lustre: Skipped 1 previous similar message [ 917.016830] Lustre: server umount lustre-OST0000 complete [ 923.105341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 923.119026] Lustre: Skipped 1 previous similar message [ 923.128165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 923.133637] Lustre: Skipped 1 previous similar message [ 928.226263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 928.232213] Lustre: Skipped 1 previous similar message [ 932.321211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 932.331270] Lustre: Skipped 1 previous similar message [ 935.392615] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 935.639449] Lustre: server umount lustre-MDT0000 complete [ 937.464105] LustreError: 32412:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 937.478348] LustreError: 32412:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 939.136042] LustreError: 29950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066788 with bad export cookie 10264483817412239068 [ 939.145718] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 939.567721] Lustre: server umount lustre-MDT0001 complete [ 949.898976] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 952.465823] Key type lgssc unregistered [ 952.755272] LNet: 33708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.765481] LNetError: 33708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.782799] LNet: Removed LNI 192.168.203.137@tcp [ 953.627858] Key type .llcrypt unregistered [ 953.630437] Key type ._llcrypt unregistered [ 971.610314] Key type ._llcrypt registered [ 971.614873] Key type .llcrypt registered [ 971.768355] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 972.802291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 972.855219] alg: No test for adler32 (adler32-zlib) [ 974.221150] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 974.627364] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 976.375498] Key type lgssc registered [ 977.637768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 986.583278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 986.599989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 988.201068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.252380] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 997.356845] LustreError: 34976:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 997.378848] LustreError: 34976:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1001.929109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1002.436253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1007.657853] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1015.343937] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1020.805948] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1027.181168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1027.548468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1032.741891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1033.160496] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1040.216027] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1049.057507] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1049.067547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1049.077799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1053.153791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1053.163319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1058.280272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1058.286811] Lustre: Skipped 1 previous similar message [ 1062.879145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1063.075667] Lustre: server umount lustre-OST0000 complete [ 1068.512580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1068.514992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1068.531951] Lustre: Skipped 1 previous similar message [ 1068.546117] Lustre: Skipped 2 previous similar messages [ 1073.415702] Lustre: server umount lustre-MDT0000 complete [ 1073.646465] LustreError: 34976:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1073.667196] LustreError: 34976:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1077.753953] LustreError: 35711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066927 with bad export cookie 15447571163549453781 [ 1077.767317] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1077.771755] LustreError: 35711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1078.316367] Lustre: server umount lustre-MDT0001 complete [ 1088.264411] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 1090.787265] Key type lgssc unregistered [ 1091.058693] LNet: 38524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.069411] LNetError: 38524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1091.090927] LNet: Removed LNI 192.168.203.137@tcp [ 1091.803741] Key type .llcrypt unregistered [ 1091.808868] Key type ._llcrypt unregistered [ 1113.547569] Key type ._llcrypt registered [ 1113.553295] Key type .llcrypt registered [ 1113.679289] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 1114.990329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1115.011027] alg: No test for adler32 (adler32-zlib) [ 1116.142205] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 1116.394620] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 1118.071335] Key type lgssc registered [ 1119.175759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1127.039623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1127.073202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1128.571115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1132.821246] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1139.173177] LustreError: 39939:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1139.184810] LustreError: 39939:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1141.154197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1141.533046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1145.333624] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1151.900449] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1156.947458] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1162.695239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1162.986375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1168.702281] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1174.261414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1175.983883] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1182.689718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1182.709859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1182.745846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1183.816965] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 1184.741021] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1184.757058] Lustre: Skipped 1 previous similar message [ 1188.913273] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 1188.928104] Lustre: Skipped 3 previous similar messages [ 1189.140943] Lustre: server umount lustre-MDT0000 complete [ 1189.858000] LustreError: 41363:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1189.878917] LustreError: 41363:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1192.552650] LustreError: 39918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067042 with bad export cookie 17971875299313529820 [ 1192.557576] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1192.558033] LustreError: 39918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1194.030901] LustreError: 39934:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1194.054464] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.37@tcp (stopping) [ 1194.977731] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1197.984065] LustreError: 39933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1198.017477] LustreError: 39933:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1199.137510] Lustre: server umount lustre-MDT0001 complete [ 1228.255211] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1228.425722] Lustre: server umount lustre-OST0000 complete [ 1243.746484] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 1246.529494] Key type lgssc unregistered [ 1246.813168] LNet: 43798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1246.822555] LNetError: 43798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1246.832809] LNet: Removed LNI 192.168.203.137@tcp [ 1247.507353] Key type .llcrypt unregistered [ 1247.509283] Key type ._llcrypt unregistered [ 1268.185856] Key type ._llcrypt registered [ 1268.189317] Key type .llcrypt registered [ 1268.299147] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 1269.092813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1269.233413] alg: No test for adler32 (adler32-zlib) [ 1270.259307] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 1270.506082] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 1272.151178] Key type lgssc registered [ 1273.069902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1279.585942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1279.601630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.112276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1284.917916] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1290.213888] LustreError: 45206:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1290.225771] LustreError: 45206:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1292.883513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1293.179130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1297.918733] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1305.647936] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1310.937292] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1316.828245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1317.142049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1323.362118] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1325.358416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1330.530250] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1338.442273] Lustre: lustre-OST0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 1338.850556] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1338.860805] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.898566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.906391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1340.919165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1340.932442] Lustre: Skipped 2 previous similar messages [ 1343.533732] Lustre: lustre-OST0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 1346.018804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1346.032424] Lustre: Skipped 1 previous similar message [ 1351.135183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1351.148348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1351.166791] Lustre: Skipped 2 previous similar messages [ 1351.309636] Lustre: server umount lustre-OST0000 complete [ 1370.080072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1370.098444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1370.122727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1374.927876] Lustre: server umount lustre-MDT0000 complete [ 1376.743712] LustreError: 45206:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1376.770717] LustreError: 45206:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1378.171038] LustreError: 45187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067228 with bad export cookie 5781791845108069381 [ 1378.179477] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1378.188961] LustreError: 45187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1384.646875] Lustre: server umount lustre-MDT0001 complete [ 1396.060042] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 1399.349731] Key type lgssc unregistered [ 1399.709903] LNet: 48914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1399.720738] LNetError: 48914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1399.747890] LNet: Removed LNI 192.168.203.137@tcp [ 1400.676686] Key type .llcrypt unregistered [ 1400.680064] Key type ._llcrypt unregistered [ 1426.682369] Key type ._llcrypt registered [ 1426.689328] Key type .llcrypt registered [ 1426.807341] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 1428.080186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1428.137727] alg: No test for adler32 (adler32-zlib) [ 1429.360936] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 1429.671148] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 1431.415297] Key type lgssc registered [ 1432.573038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1440.526281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1440.564555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1442.055150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1447.469025] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1451.494347] LustreError: 50336:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1451.523555] LustreError: 50336:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1456.303917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1456.609570] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1456.779726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1461.974809] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1469.481148] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1476.227790] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1482.642917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1483.013450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1487.118396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1490.186800] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1500.360673] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1505.515679] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1508.408068] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1520.937367] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-1015h09m28s [ 1522.365735] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1527.266616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1527.271337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1527.281838] Lustre: Skipped 2 previous similar messages [ 1527.287314] Lustre: Skipped 1 previous similar message [ 1528.357020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1528.361354] Lustre: Skipped 2 previous similar messages [ 1530.450872] Lustre: server umount lustre-MDT0000 complete [ 1533.391295] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1534.009582] LustreError: 50331:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1538.623870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1538.733509] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1538.929209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1538.986535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1539.947927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1542.479534] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1545.366646] Lustre: DEBUG MARKER: Wait for df (36686) ... [ 1546.675723] Lustre: DEBUG MARKER: done [ 1552.864294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1552.876422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1552.894780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1552.897497] Lustre: Skipped 2 previous similar messages [ 1554.400290] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1554.410705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1556.708606] Lustre: server umount lustre-OST0000 complete [ 1560.544575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1560.546720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1560.553371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1560.581922] Lustre: Skipped 1 previous similar message [ 1560.587116] Lustre: Skipped 3 previous similar messages [ 1566.465687] Lustre: server umount lustre-MDT0000 complete [ 1566.688187] LustreError: 50335:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1566.699204] LustreError: 50335:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1569.214351] LustreError: 50318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067419 with bad export cookie 6702582555109094726 [ 1569.218797] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1569.223241] LustreError: 50318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1569.462373] Lustre: server umount lustre-MDT0001 complete [ 1577.567934] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 1580.354594] Key type lgssc unregistered [ 1580.604261] LNet: 55261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1580.611588] LNetError: 55261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1580.625656] LNet: Removed LNI 192.168.203.137@tcp [ 1581.475771] Key type .llcrypt unregistered [ 1581.478321] Key type ._llcrypt unregistered [ 1621.242697] Key type ._llcrypt registered [ 1621.244634] Key type .llcrypt registered [ 1621.376848] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 1622.516334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1622.698235] alg: No test for adler32 (adler32-zlib) [ 1623.710331] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 1623.932589] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 1625.671231] Key type lgssc registered [ 1626.993619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1634.618695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1634.642409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1635.881169] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1635.895770] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1636.037782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1640.128590] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1648.658451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1648.786841] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1648.847613] Lustre: 57469:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1649.202525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1653.465647] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1659.917662] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1664.415461] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1670.029247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1670.208402] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1670.393363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1675.528169] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1676.916260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1682.480886] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1687.996871] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-1015h12m16s [ 1689.553895] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1695.142703] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1697.403874] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1697.429662] LustreError: 59693:0:(mdt_handler.c:2397:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1717.836096] Lustre: lustre-MDT0000: Client 48eb6daa-431c-4a11-97a3-7719839dfeef (at 192.168.203.37@tcp) reconnecting [ 1737.455174] LustreError: 59693:0:(mdt_handler.c:2397:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1739.201548] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1741.471458] Lustre: DEBUG MARKER: done [ 1746.918321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.921677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1746.942536] Lustre: Skipped 1 previous similar message [ 1746.951698] Lustre: Skipped 1 previous similar message [ 1752.034399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1752.038981] Lustre: Skipped 1 previous similar message [ 1752.407556] Lustre: server umount lustre-OST0000 complete [ 1756.646244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1756.651857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1756.660724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1762.419241] Lustre: server umount lustre-MDT0000 complete [ 1765.535746] LustreError: 57461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067615 with bad export cookie 16555112041012254519 [ 1765.538826] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1765.542896] LustreError: 57461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1765.812193] Lustre: server umount lustre-MDT0001 complete [ 1774.860897] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 1777.047438] Key type lgssc unregistered [ 1777.309398] LNet: 61793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1777.317424] LNetError: 61793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1777.338860] LNet: Removed LNI 192.168.203.137@tcp [ 1778.074385] Key type .llcrypt unregistered [ 1778.081788] Key type ._llcrypt unregistered [ 1810.272680] Key type ._llcrypt registered [ 1810.275681] Key type .llcrypt registered [ 1810.385946] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 1810.983610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1811.033480] alg: No test for adler32 (adler32-zlib) [ 1812.019441] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 1812.212199] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 1813.879560] Key type lgssc registered [ 1814.757207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1817.269955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1817.293392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1818.535614] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1818.573683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1818.700975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1819.477103] Lustre: Failing over lustre-MDT0000 [ 1819.703947] Lustre: server umount lustre-MDT0000 complete [ 1824.343632] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 15:14:33 (1789067673) [ 1829.255144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1829.570496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1833.011213] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1839.434464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1839.522555] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1839.555516] Lustre: 64564:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1839.764623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1843.058738] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1848.933414] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1853.531345] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1858.660779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1858.846627] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1858.983156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1864.407151] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1867.402443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1870.531868] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1876.195676] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1883.110017] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1883.122324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1883.134269] Lustre: Skipped 2 previous similar messages [ 1883.143926] Lustre: Skipped 2 previous similar messages [ 1888.120433] Lustre: server umount lustre-MDT0000 complete [ 1890.902384] LustreError: 64556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067740 with bad export cookie 1867915788640182783 [ 1890.908167] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1890.913145] LustreError: 64556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1891.247555] Lustre: server umount lustre-MDT0001 complete [ 1894.143812] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1897.784906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1905.595492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1909.727417] Lustre: 63488:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789067743/real 1789067743] req@00000000ce7552b4 x1875973412785792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789067759 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1909.752687] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.776940] Lustre: 63487:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789067745/real 1789067745] req@00000000f40ee9c0 x1875973412785920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789067761 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1914.847250] Lustre: 63486:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789067748/real 1789067748] req@00000000d0286890 x1875973412786048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789067764 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1915.950710] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x19ec2b835c3deae2 [ 1915.964571] Lustre: MGC192.168.203.137@tcp: Connection restored to 0@lo (at 0@lo) [ 1916.173290] LustreError: 66421:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1916.197870] LustreError: 66421:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1916.259199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1916.341473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1917.217324] Lustre: 63486:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789067750/real 1789067750] req@0000000048e0f3d1 x1875973412786304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789067766 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1918.307238] LustreError: 66422:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1919.330878] LustreError: 66421:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1919.346580] LustreError: 66421:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1920.378176] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1921.375110] Lustre: 63487:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789067755/real 1789067755] req@000000009473d15a x1875973412786432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789067771 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1924.589268] LustreError: 68342:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1924.602737] LustreError: 68342:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1927.657132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1927.990806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1931.870247] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1933.295556] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1938.429605] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1938.498925] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1943.637343] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1950.098580] Lustre: server umount lustre-MDT0000 complete [ 1953.238285] LustreError: 64556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067803 with bad export cookie 1867915788640185058 [ 1953.245499] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1953.253820] LustreError: 64556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1953.765254] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.779251] LustreError: 68341:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1953.782430] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1955.561736] Lustre: server umount lustre-MDT0001 complete [ 1961.529368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1969.862124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1978.604426] LustreError: 66421:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1978.668804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1978.712780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1982.218707] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1989.142319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1993.590799] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1994.165188] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2001.837603] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2007.432166] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2014.191875] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.202664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2014.210954] Lustre: Skipped 2 previous similar messages [ 2014.231054] Lustre: Skipped 2 previous similar messages [ 2019.301272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2019.313268] Lustre: Skipped 2 previous similar messages [ 2020.238910] Lustre: server umount lustre-MDT0000 complete [ 2024.397592] LustreError: 64555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067874 with bad export cookie 1867915788640186437 [ 2024.418135] LustreError: 64555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2024.419800] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2024.430303] LustreError: 71023:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2024.430313] LustreError: 71023:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2024.430810] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2024.475521] Lustre: Skipped 1 previous similar message [ 2024.479803] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2024.786721] Lustre: server umount lustre-MDT0001 complete [ 2032.372401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2035.932797] Lustre: DEBUG MARKER: files compared the same [ 2038.141520] Lustre: server umount lustre-OST0000 complete [ 2055.267689] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2058.249676] Key type lgssc unregistered [ 2058.534715] LNet: 74293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2058.546305] LNetError: 74293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2059.621961] LNet: Removed LNI 192.168.203.137@tcp [ 2060.830234] Key type .llcrypt unregistered [ 2060.833336] Key type ._llcrypt unregistered [ 2088.711912] Key type ._llcrypt registered [ 2088.713827] Key type .llcrypt registered [ 2088.879937] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2090.312829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2090.434582] alg: No test for adler32 (adler32-zlib) [ 2091.614272] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 2091.937724] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2093.657188] Key type lgssc registered [ 2094.796634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2103.020236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2103.048494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2104.585427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2108.970746] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug +malloc all [ 2115.040609] LustreError: 75709:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2115.080578] LustreError: 75709:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2117.949205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2118.371327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2122.537966] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug +malloc all [ 2129.293837] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2134.601430] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2140.864335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2141.165099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2147.520525] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug +malloc all [ 2149.369676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2154.782963] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2163.679965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2163.689442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2163.705585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2164.708010] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2164.708472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2164.720752] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.792237] Lustre: server umount lustre-OST0000 complete [ 2174.946124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2174.948941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2174.956712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2174.956720] Lustre: Skipped 1 previous similar message [ 2174.993228] Lustre: Skipped 1 previous similar message [ 2180.069570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2180.081802] Lustre: Skipped 2 previous similar messages [ 2185.183971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2185.188965] Lustre: Skipped 1 previous similar message [ 2186.208834] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2186.411731] Lustre: server umount lustre-MDT0000 complete [ 2189.288247] LustreError: 75709:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2189.319872] LustreError: 75709:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2189.456629] LustreError: 75689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068039 with bad export cookie 11626613608761911836 [ 2189.458888] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2189.474964] LustreError: 75689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2189.744046] Lustre: server umount lustre-MDT0001 complete [ 2200.391349] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2203.631845] Key type lgssc unregistered [ 2203.953899] LNet: 79243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2203.962024] LNetError: 79243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2203.981738] LNet: Removed LNI 192.168.203.137@tcp [ 2205.018726] Key type .llcrypt unregistered [ 2205.023764] Key type ._llcrypt unregistered [ 2229.229990] Key type ._llcrypt registered [ 2229.233133] Key type .llcrypt registered [ 2229.342699] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2230.840720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2230.904248] alg: No test for adler32 (adler32-zlib) [ 2232.063784] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 2232.411633] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2234.103216] Key type lgssc registered [ 2235.192967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2242.477917] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2242.498912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2258.079403] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2263.199501] LustreError: 80627:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.137@tcp: failed processing log, type 4: rc = -110 [ 2289.823189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2295.356755] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2301.824920] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2308.457310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.568201] Lustre: *** cfs_fail_loc=706, val=0*** [ 2308.914159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2308.998551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2313.276689] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2314.209063] LustreError: 80652:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.218348] LustreError: 80652:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2319.334676] LustreError: 80653:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.059061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2320.364194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2324.265452] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2330.195450] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2335.085286] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2339.817801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2339.833646] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2339.866284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2339.886120] Lustre: Skipped 1 previous similar message [ 2344.931660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2350.053774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2350.062651] Lustre: Skipped 2 previous similar messages [ 2351.074298] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2351.276276] Lustre: server umount lustre-OST0000 complete [ 2355.173194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2355.192980] Lustre: Skipped 1 previous similar message [ 2355.201926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2355.210514] Lustre: Skipped 1 previous similar message [ 2360.289205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2360.305777] Lustre: Skipped 1 previous similar message [ 2360.739110] Lustre: server umount lustre-MDT0000 complete [ 2364.571797] LustreError: 80633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068214 with bad export cookie 1351071462277323554 [ 2364.591366] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2365.282033] Lustre: server umount lustre-MDT0001 complete [ 2377.557668] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2380.590751] Key type lgssc unregistered [ 2380.930764] LNet: 84240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2380.936137] LNetError: 84240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2380.951253] LNet: Removed LNI 192.168.203.137@tcp [ 2381.936324] Key type .llcrypt unregistered [ 2381.941850] Key type ._llcrypt unregistered [ 2405.468455] Key type ._llcrypt registered [ 2405.472466] Key type .llcrypt registered [ 2405.607343] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2406.313189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2406.327242] alg: No test for adler32 (adler32-zlib) [ 2407.571953] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 2407.964194] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2409.655809] Key type lgssc registered [ 2410.755253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2417.154918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2417.179019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2422.049671] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2428.609946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2429.077859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2433.907873] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2434.529046] LustreError: 86353:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2434.570466] LustreError: 86353:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2438.626961] LustreError: 86357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2438.657310] LustreError: 86357:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2442.362891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2442.604241] LustreError: 86353:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2442.628521] LustreError: 86353:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2442.644814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2447.840505] LustreError: 86356:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.860111] LustreError: 86356:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2448.804451] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2452.967665] LustreError: 86357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.988231] LustreError: 86357:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2456.377375] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2456.384974] Lustre: MGS: Client 90f90f9b-c3e2-4fd0-bf0a-88993b4f79bc (at 0@lo) reconnecting [ 2456.398783] LustreError: 86367:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2456.652693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2456.723382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2460.542123] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2477.038386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2477.044560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2477.050606] Lustre: Skipped 1 previous similar message [ 2477.056509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2477.058468] Lustre: Skipped 1 previous similar message [ 2479.539845] Lustre: server umount lustre-OST0000 complete [ 2487.264362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2487.277961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2487.279970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2487.297718] Lustre: Skipped 1 previous similar message [ 2487.316918] Lustre: Skipped 1 previous similar message [ 2488.642281] Lustre: server umount lustre-MDT0000 complete [ 2492.560572] Lustre: server umount lustre-MDT0001 complete [ 2496.047438] Lustre: server umount MGS complete [ 2507.846343] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 15:25:55 (1789068355) [ 2539.063424] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 2548.158596] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2582.146206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2586.575160] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2596.255587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2596.474521] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2596.515825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2596.589383] Lustre: lustre-MDT0001: new disk, initializing [ 2596.669501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2596.701247] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2596.711083] Lustre: 93522:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2596.716839] LustreError: 93522:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2600.652624] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2611.003718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2611.344737] Lustre: lustre-OST0000: new disk, initializing [ 2611.347837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2611.353708] Lustre: 94289:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2611.409885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2616.701420] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2621.472879] Lustre: 94806:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004805e330 x1875974036691968/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789068487 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2625.240653] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2632.266607] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2632.279380] Lustre: MGS: Client 883c79ad-a326-46a4-ad11-f95e56d004c8 (at 0@lo) reconnecting [ 2632.330297] Lustre: 95334:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2632.348830] LustreError: 93508:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2632.360054] LustreError: 93508:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2632.379943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2632.459769] Lustre: lustre-MDT0000: new disk, initializing [ 2632.599377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2632.618639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2632.765735] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2633.696708] Lustre: 95358:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000000741696b x1875974036698496/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789068499 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2637.308666] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2638.829205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2638.839307] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2638.982603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2662.368674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2662.374142] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2662.385488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2664.418681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2664.420424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2667.493826] Lustre: server umount lustre-OST0000 complete [ 2674.661160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2674.661702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2674.672359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2674.672368] Lustre: Skipped 1 previous similar message [ 2674.732179] Lustre: Skipped 1 previous similar message [ 2677.789079] Lustre: server umount lustre-MDT0000 complete [ 2679.786155] LustreError: 93497:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2679.808502] LustreError: 93497:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2687.963610] Lustre: server umount lustre-MDT0001 complete [ 2692.092615] Lustre: server umount MGS complete [ 2703.324322] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 15:29:11 (1789068551) [ 2719.738926] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2722.775753] Key type lgssc unregistered [ 2723.114225] LNet: 97854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2723.128615] LNetError: 97854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2723.153439] LNet: Removed LNI 192.168.203.137@tcp [ 2724.189320] Key type .llcrypt unregistered [ 2724.192869] Key type ._llcrypt unregistered [ 2742.621328] Key type ._llcrypt registered [ 2742.625397] Key type .llcrypt registered [ 2742.750898] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2743.815851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2743.840741] alg: No test for adler32 (adler32-zlib) [ 2745.198574] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 2745.500802] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2747.287402] Key type lgssc registered [ 2749.175844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2758.063931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2758.097328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2759.745923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2765.064196] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2769.892799] LustreError: 99076:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.911880] LustreError: 99076:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2774.059234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2774.524909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2779.706719] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2787.053068] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2792.627698] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2799.550716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2799.778607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2804.931346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2805.934922] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2814.168819] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2824.606219] Lustre: server umount lustre-OST0000 complete [ 2830.818752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2830.824267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2830.856986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2835.937657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2835.953343] Lustre: Skipped 2 previous similar messages [ 2839.073990] Lustre: server umount lustre-MDT0000 complete [ 2841.059109] LustreError: 99076:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2842.080140] Lustre: 98386:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789068675/real 1789068675] req@0000000044e22ea1 x1875974391096576/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789068691 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2842.137164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2842.157726] Lustre: Skipped 1 previous similar message [ 2842.171675] LustreError: 100053:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2842.189050] LustreError: 100053:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2842.885598] LustreError: 101654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068692 with bad export cookie 15983443646968362551 [ 2842.889153] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2842.895561] LustreError: 101654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2843.305897] Lustre: server umount lustre-MDT0001 complete [ 2855.456547] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2858.318423] Key type lgssc unregistered [ 2858.642064] LNet: 102606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2858.649552] LNetError: 102606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2858.675602] LNet: Removed LNI 192.168.203.137@tcp [ 2859.771110] Key type .llcrypt unregistered [ 2859.777373] Key type ._llcrypt unregistered [ 2876.590490] Key type ._llcrypt registered [ 2876.596161] Key type .llcrypt registered [ 2876.747212] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2877.767129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2877.794051] alg: No test for adler32 (adler32-zlib) [ 2878.881902] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 2879.139832] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2880.844152] Key type lgssc registered [ 2882.292859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2888.821632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2888.828635] LustreError: 103674:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2888.917951] LustreError: 103686:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2888.925840] LustreError: 103674:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4908 [ 2888.937361] LustreError: 103674:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2888.949966] LustreError: 103674:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2888.959766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.604786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2898.009168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2898.400705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2903.368259] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2908.505414] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing lsmod [ 2911.638435] LustreError: 104890:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2911.657675] LustreError: 104891:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2911.669797] LustreError: 104890:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4979 [ 2911.678783] LustreError: 104890:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2911.690599] LustreError: 104890:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2911.738779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2912.180819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2917.346713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2917.360980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2917.371535] Lustre: Skipped 2 previous similar messages [ 2917.388827] Lustre: Skipped 1 previous similar message [ 2921.459225] Lustre: server umount lustre-MDT0000 complete [ 2922.473832] LustreError: 103718:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2922.507035] LustreError: 103718:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2923.490855] LustreError: 103723:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2923.509401] LustreError: 103723:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2924.999114] LustreError: 103700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068774 with bad export cookie 15846449390345289035 [ 2925.013260] LustreError: 103700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2925.014444] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2925.507592] Lustre: server umount lustre-MDT0001 complete [ 2938.429919] Lustre: server umount lustre-OST0000 complete [ 2946.276713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2946.797098] LustreError: 105954:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2946.897701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2951.093849] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2952.173551] LustreError: 105955:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2956.264225] LustreError: 105954:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2959.333491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2959.723829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2964.310840] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2972.844338] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2978.712111] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2984.895679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2985.263065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2991.124807] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2991.409655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2999.028709] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3012.066302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3012.076855] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3012.102382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3012.114494] Lustre: Skipped 1 previous similar message [ 3015.138304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3015.140185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3015.162618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3017.888049] Lustre: server umount lustre-OST0000 complete [ 3021.290293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3021.298695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.309883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3021.313421] Lustre: Skipped 1 previous similar message [ 3025.376877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3025.377689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3025.405289] Lustre: Skipped 1 previous similar message [ 3027.460454] Lustre: server umount lustre-MDT0000 complete [ 3030.496541] LustreError: 105951:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3030.515806] LustreError: 105951:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3030.676584] LustreError: 105934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068880 with bad export cookie 15846449390345289749 [ 3030.684560] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3030.686059] LustreError: 105934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3031.033510] Lustre: server umount lustre-MDT0001 complete [ 3041.076849] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 3043.645888] Key type lgssc unregistered [ 3044.012089] LNet: 109507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3044.017915] LNetError: 109507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3044.060220] LNet: Removed LNI 192.168.203.137@tcp [ 3044.940964] Key type .llcrypt unregistered [ 3044.947267] Key type ._llcrypt unregistered [ 3065.704297] Key type ._llcrypt registered [ 3065.712993] Key type .llcrypt registered [ 3065.866399] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 3066.767510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3066.819555] alg: No test for adler32 (adler32-zlib) [ 3067.857255] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 3068.060127] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 3069.759273] Key type lgssc registered [ 3071.090290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3079.167524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3079.179130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3080.693391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3084.962685] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3090.933086] LustreError: 110902:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3090.954771] LustreError: 110902:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3093.280067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3093.655361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3098.122954] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3105.279215] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3110.555617] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3116.775182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3117.157662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3123.073632] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3125.365394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3130.549569] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3135.049215] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3138.209465] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3144.060357] Lustre: server umount lustre-OST0000 complete [ 3150.308355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3150.322320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3150.339994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3151.329232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3151.332073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3156.453948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3156.468393] Lustre: Skipped 2 previous similar messages [ 3159.519296] Lustre: 110214:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789068993/real 1789068993] req@000000002d68b446 x1875974728746368/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789069009 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3159.553287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3159.581347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3159.586181] Lustre: Skipped 1 previous similar message [ 3162.079130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3162.207789] Lustre: server umount lustre-MDT0000 complete [ 3162.593841] Lustre: 110215:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789068996/real 1789068996] req@00000000b5a5cdcd x1875974728747264/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789069012 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3164.641542] LustreError: 110902:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3164.679209] LustreError: 110902:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3166.037454] LustreError: 110884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069015 with bad export cookie 7142083828691081083 [ 3166.043748] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3166.048260] LustreError: 110884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3166.386878] Lustre: server umount lustre-MDT0001 complete [ 3176.762793] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 3179.781607] Key type lgssc unregistered [ 3180.229792] LNet: 114541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3180.258048] LNetError: 114541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3180.302287] LNet: Removed LNI 192.168.203.137@tcp [ 3181.496868] Key type .llcrypt unregistered [ 3181.499526] Key type ._llcrypt unregistered [ 3202.077365] Key type ._llcrypt registered [ 3202.081590] Key type .llcrypt registered [ 3202.177358] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 3203.521839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3203.576701] alg: No test for adler32 (adler32-zlib) [ 3204.595438] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 3204.797621] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 3206.487919] Key type lgssc registered [ 3207.572581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3215.987877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3216.016434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3217.849624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3222.619353] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3228.136367] LustreError: 115763:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3228.159340] LustreError: 115763:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3231.249575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3231.743026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3236.212613] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3243.705144] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3249.056555] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3255.686159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3256.128112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3262.053496] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3262.330575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3269.402923] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3282.913983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3282.924028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3282.942915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3282.948298] Lustre: Skipped 1 previous similar message [ 3284.262760] Lustre: server umount lustre-OST0000 complete [ 3293.153263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3293.158345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3293.180047] LustreError: Skipped 1 previous similar message [ 3293.181422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3293.229917] Lustre: Skipped 2 previous similar messages [ 3294.774765] Lustre: server umount lustre-MDT0000 complete [ 3298.273881] LustreError: 115763:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3298.286467] LustreError: 115763:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3299.318474] LustreError: 115744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069149 with bad export cookie 6963504893100171834 [ 3299.331468] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3299.336979] LustreError: 115744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3299.786376] Lustre: server umount lustre-MDT0001 complete [ 3311.327451] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 3314.728885] Key type lgssc unregistered [ 3315.049314] LNet: 119296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3315.063725] LNetError: 119296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3315.098296] LNet: Removed LNI 192.168.203.137@tcp [ 3315.864979] Key type .llcrypt unregistered [ 3315.866947] Key type ._llcrypt unregistered [ 3339.004639] Key type ._llcrypt registered [ 3339.007051] Key type .llcrypt registered [ 3339.118922] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 3340.232161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3340.300735] alg: No test for adler32 (adler32-zlib) [ 3341.361440] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 3341.585810] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 3343.359331] Key type lgssc registered [ 3344.771089] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3353.788420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3353.816593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3355.490516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3361.426870] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3365.866248] LustreError: 120710:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3365.893239] LustreError: 120710:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3370.341887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3370.804706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3375.448640] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3382.829995] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3389.156902] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3397.491790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3398.088252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3401.152357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3405.429127] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3413.989156] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3420.890222] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3424.235561] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3424.249213] Lustre: Skipped 2 previous similar messages [ 3428.469697] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3428.528684] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3429.197421] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3429.201172] Lustre: Skipped 2 previous similar messages [ 3434.568467] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3434.572728] Lustre: Skipped 1 previous similar message [ 3439.710898] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.37@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3439.723624] Lustre: Skipped 2 previous similar messages [ 3440.027060] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.37@tcp 192.168.0.[2,10]@tcp [ 3451.202426] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.37/24@tcp in log lustre-MDT0000 [ 3451.211835] Lustre: Skipped 2 previous similar messages [ 3460.114530] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.37/24@tcp [ 3460.120365] Lustre: Skipped 1 previous similar message [ 3466.726557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3466.738169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3466.741825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3466.746498] Lustre: Skipped 1 previous similar message [ 3470.611926] Lustre: server umount lustre-OST0000 complete [ 3476.966718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3476.970285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3476.994431] Lustre: Skipped 1 previous similar message [ 3476.997382] Lustre: Skipped 1 previous similar message [ 3482.085650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3482.097544] Lustre: Skipped 2 previous similar messages [ 3486.182463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3486.195483] Lustre: Skipped 1 previous similar message [ 3488.223276] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3488.550873] Lustre: server umount lustre-MDT0000 complete [ 3491.304708] LustreError: 120710:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3491.349971] LustreError: 120710:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3492.531693] LustreError: 120689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069342 with bad export cookie 3369990898069455907 [ 3492.545066] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3492.552459] LustreError: 120689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3493.189365] Lustre: server umount lustre-MDT0001 complete [ 3504.574675] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 3507.433928] Key type lgssc unregistered [ 3507.740750] LNet: 125659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3507.750255] LNetError: 125659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3508.775275] LNet: Removed LNI 192.168.203.137@tcp [ 3509.853910] Key type .llcrypt unregistered [ 3509.858178] Key type ._llcrypt unregistered [ 3542.162402] Key type ._llcrypt registered [ 3542.164857] Key type .llcrypt registered [ 3542.316950] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 3544.053510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3544.146929] alg: No test for adler32 (adler32-zlib) [ 3545.416745] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 3545.693187] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 3547.431193] Key type lgssc registered [ 3549.097422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3557.313858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3557.328587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3558.728507] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3558.745303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3558.747666] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.37/24@tcp [ 3563.470661] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3569.123514] LustreError: 127220:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3569.139690] LustreError: 127220:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3571.489929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3571.816272] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3571.863841] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.37/24@tcp [ 3571.918551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3576.244734] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3582.990203] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3588.619009] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3595.080123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3595.492590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3603.316906] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3603.718156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3614.293886] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3624.433360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3624.440629] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3624.446670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3624.463489] Lustre: Skipped 1 previous similar message [ 3629.543116] Lustre: server umount lustre-OST0000 complete [ 3633.634440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3633.641298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3633.659184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3633.666962] Lustre: Skipped 1 previous similar message [ 3634.670645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3634.673166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3639.759852] Lustre: server umount lustre-MDT0000 complete [ 3639.780155] LustreError: 127216:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3639.805358] LustreError: 127216:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3644.253912] LustreError: 127199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069494 with bad export cookie 16174818907006491178 [ 3644.270385] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3644.288498] LustreError: 127199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3644.893846] Lustre: server umount lustre-MDT0001 complete [ 3659.930674] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 3664.498576] Key type lgssc unregistered [ 3665.002199] LNet: 130807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3665.020614] LNetError: 130807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3665.039929] LNet: Removed LNI 192.168.203.137@tcp [ 3666.400910] Key type .llcrypt unregistered [ 3666.411735] Key type ._llcrypt unregistered [ 3694.640539] Key type ._llcrypt registered [ 3694.647767] Key type .llcrypt registered [ 3694.764729] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 3713.983881] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 3715.710772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3715.841473] alg: No test for adler32 (adler32-zlib) [ 3717.332283] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 3717.741449] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 3719.519204] Key type lgssc registered [ 3720.908397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3763.408527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3763.434579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3764.648403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3764.676914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3764.747974] Lustre: lustre-MDT0000: new disk, initializing [ 3764.797800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3764.830986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3769.305574] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3780.893231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3781.037493] Lustre: 134209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3781.138885] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3781.150761] Lustre: Skipped 1 previous similar message [ 3781.283457] Lustre: lustre-MDT0001: new disk, initializing [ 3781.417793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3781.471380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3781.485868] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3786.009621] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3795.420642] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3801.010170] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3807.831334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3808.211640] Lustre: lustre-OST0000: new disk, initializing [ 3808.221344] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3808.236291] Lustre: 136174:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3808.327811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3814.986039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3815.006145] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3815.117115] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3815.677368] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3824.729863] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3829.018567] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3829.281721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3834.264688] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3834.496208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3840.993821] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3840.996771] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3841.002863] LustreError: Skipped 1 previous similar message [ 3841.010488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3841.015490] Lustre: Skipped 1 previous similar message [ 3841.036446] Lustre: Skipped 1 previous similar message [ 3842.611434] Lustre: server umount lustre-OST0000 complete [ 3846.120758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3846.130340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3846.130649] Lustre: Skipped 1 previous similar message [ 3846.152480] Lustre: Skipped 1 previous similar message [ 3851.233659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3851.247333] Lustre: Skipped 1 previous similar message [ 3852.438626] Lustre: server umount lustre-MDT0000 complete [ 3856.511779] LustreError: 134200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069706 with bad export cookie 14214035580351337407 [ 3856.523849] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3856.530493] LustreError: 134200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3856.953451] Lustre: server umount lustre-MDT0001 complete [ 3866.031453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3866.761967] LustreError: 138394:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3866.793575] LustreError: 138394:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3866.917879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3872.233529] LustreError: 138395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3872.526492] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3876.332715] LustreError: 138394:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3881.440629] LustreError: 138395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3882.414589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3882.968746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3889.496176] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3899.803398] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3906.168149] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3912.783657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3913.280396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3920.533181] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3928.871956] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3933.025706] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3937.617543] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3942.466606] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3942.701841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3951.680032] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3951.844364] Lustre: lustre-OST0001: new disk, initializing [ 3951.849423] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3951.852993] Lustre: lustre-OST0001: Not available for connect from 192.168.203.37@tcp (not set up) [ 3951.862047] Lustre: 141795:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3951.951278] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3957.858820] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3960.975482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3961.018726] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3961.136829] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3966.726765] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3966.933809] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3971.530481] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3971.755535] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3977.891317] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88c7891ac800.ost_server_uuid 50 [ 3979.523723] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88c7891ac800.ost_server_uuid in IDLE state after 0 sec [ 3989.874376] Lustre: server umount lustre-OST0001 complete [ 3992.032857] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3992.047197] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3992.063439] LustreError: 140216:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3992.073494] LustreError: 140216:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3994.080342] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3994.091477] LustreError: Skipped 1 previous similar message [ 3994.100668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3994.123548] Lustre: Skipped 1 previous similar message [ 3994.132826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4000.024344] Lustre: server umount lustre-OST0000 complete [ 4005.856959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4005.866695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4005.884609] Lustre: Skipped 1 previous similar message [ 4005.899352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4005.917685] Lustre: Skipped 2 previous similar messages [ 4018.144321] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4018.455868] Lustre: server umount lustre-MDT0000 complete [ 4022.238043] LustreError: 138375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069872 with bad export cookie 14214035580351338289 [ 4022.241026] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4022.253315] LustreError: 138375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4022.654551] Lustre: server umount lustre-MDT0001 complete [ 4042.926559] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 4045.795810] Key type lgssc unregistered [ 4046.073173] LNet: 144798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4046.088108] LNetError: 144798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4046.103789] LNet: Removed LNI 192.168.203.137@tcp [ 4046.979842] Key type .llcrypt unregistered [ 4046.982579] Key type ._llcrypt unregistered [ 4085.119608] Key type ._llcrypt registered [ 4085.123455] Key type .llcrypt registered [ 4085.231055] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 4101.891223] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 4103.511629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4103.564423] alg: No test for adler32 (adler32-zlib) [ 4104.697816] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 4104.966730] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 4106.647179] Key type lgssc registered [ 4108.083887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4147.032089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4147.103387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4148.504561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4148.538777] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4148.725165] Lustre: lustre-MDT0000: new disk, initializing [ 4148.830530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4148.857824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4153.526348] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4167.222978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4167.349352] Lustre: 148949:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4167.432124] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4167.438479] Lustre: Skipped 1 previous similar message [ 4167.568360] Lustre: lustre-MDT0001: new disk, initializing [ 4167.727867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4167.782599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4167.795787] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4172.735282] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4183.594799] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4189.893686] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4197.493035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4197.812478] Lustre: lustre-OST0000: new disk, initializing [ 4197.818323] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4197.825505] Lustre: 150921:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4197.956398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4203.661868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4203.674802] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4203.805979] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4204.814889] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4214.757033] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4220.099960] Lustre: Failing over lustre-OST0000 [ 4220.285543] Lustre: server umount lustre-OST0000 complete [ 4237.793053] Lustre: 147174:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789070071/real 1789070071] req@00000000a2cc2fd7 x1875975816120576/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789070087 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4237.825780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4239.824248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4240.196307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4240.204918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4240.242231] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4241.247112] Lustre: 147174:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789070074/real 1789070074] req@0000000080834810 x1875975816121728/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789070090 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4241.293050] Lustre: 147174:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4242.215812] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4245.282690] Lustre: 147175:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789070079/real 1789070079] req@000000006467a068 x1875975816122880/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789070095 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4245.348022] Lustre: 147175:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4246.785655] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4249.166924] Lustre: lustre-OST0000: Recovery over after 0:07, of 3 clients 3 recovered and 0 were evicted. [ 4249.167562] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4250.207157] Lustre: 147176:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789070084/real 1789070084] req@00000000f1b371a7 x1875975816123904/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789070100 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4250.234185] Lustre: 147176:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4250.399530] Lustre: Failing over lustre-MDT0000 [ 4250.705691] Lustre: server umount lustre-MDT0000 complete [ 4252.655403] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4252.676146] Lustre: Skipped 2 previous similar messages [ 4257.760551] LustreError: 148962:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4257.784693] LustreError: 148962:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4259.372628] LustreError: 148956:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4262.897941] LustreError: 148963:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4262.920753] LustreError: 148963:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4268.003671] LustreError: 150035:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4268.040967] LustreError: 150035:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4269.087749] Lustre: 147175:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789070102/real 1789070102] req@00000000a36f885a x1875975816133888/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789070118 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4269.112241] Lustre: 147175:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4269.122697] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4269.365131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4269.485768] Lustre: MGS: Not available for connect from 192.168.203.37@tcp (not set up) [ 4274.151859] LustreError: 150035:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.165907] LustreError: 150035:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 4278.243534] LustreError: 147172:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e32820a6 x1875975816139392/t0(0) o250->MGC192.168.203.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4278.585317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4278.646755] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4279.854692] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4283.279442] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4283.889493] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4283.894782] Lustre: Skipped 1 previous similar message [ 4283.915036] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4283.952564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4292.067621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4292.071299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4292.085754] Lustre: Skipped 1 previous similar message [ 4292.096348] Lustre: Skipped 1 previous similar message [ 4297.185090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4297.192884] Lustre: Skipped 1 previous similar message [ 4302.306659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4302.318321] Lustre: Skipped 1 previous similar message [ 4305.375970] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4305.542181] Lustre: server umount lustre-OST0000 complete [ 4309.472803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4309.483051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4309.502849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4315.530610] Lustre: server umount lustre-MDT0000 complete [ 4317.667330] LustreError: 148958:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4317.686917] LustreError: 148958:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4319.248656] LustreError: 149746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789070169 with bad export cookie 8836229046945210971 [ 4319.261374] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4319.266920] LustreError: 149746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4319.675615] Lustre: server umount lustre-MDT0001 complete [ 4329.718912] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 4332.739951] Key type lgssc unregistered [ 4333.100871] LNet: 154578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4333.107796] LNetError: 154578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4333.123794] LNet: Removed LNI 192.168.203.137@tcp [ 4333.935761] Key type .llcrypt unregistered [ 4333.940984] Key type ._llcrypt unregistered [ 4370.554850] Key type ._llcrypt registered [ 4370.559212] Key type .llcrypt registered [ 4370.694642] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 4372.152383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4372.192982] alg: No test for adler32 (adler32-zlib) [ 4373.406208] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 4373.800745] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 4375.511265] Key type lgssc registered [ 4376.828820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4387.154382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4387.183084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4388.875369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4394.031714] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4399.078668] LustreError: 156095:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4399.098281] LustreError: 156095:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4403.809292] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4404.194475] LustreError: 156096:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4408.176188] LustreError: 156124:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4408.184234] LustreError: 156124:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4408.188971] LustreError: 156124:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4408.511687] Lustre: server umount lustre-MDT0000 complete [ 4424.354068] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 4434.520245] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 4467.129166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4467.355567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4467.375428] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4467.454620] Lustre: lustre-MDT0000: new disk, initializing [ 4467.519319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4467.537462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4472.328857] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4482.368577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4482.495796] Lustre: 159428:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4482.548908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4482.553025] Lustre: Skipped 1 previous similar message [ 4482.634539] Lustre: lustre-MDT0001: new disk, initializing [ 4482.761286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4482.807589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4482.821756] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4487.276564] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4496.909321] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4502.904732] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4508.628569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4508.847923] Lustre: lustre-OST0000: new disk, initializing [ 4508.852132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4508.858963] Lustre: 161396:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4508.950384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4514.913793] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4517.980699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4518.000949] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4518.133661] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4524.035422] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4528.019978] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4529.823841] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 15:59:38 (1789070378) [ 4759.007648] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4759.012417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4759.035252] Lustre: Skipped 1 previous similar message [ 4764.129416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4764.137976] Lustre: Skipped 4 previous similar messages [ 4769.248888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4769.256966] Lustre: Skipped 2 previous similar messages [ 4773.343198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4773.704153] Lustre: server umount lustre-MDT0000 complete [ 4774.380461] LustreError: 159440:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4774.407526] LustreError: 159440:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4777.664317] LustreError: 159420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789070627 with bad export cookie 5521385910175760977 [ 4777.672152] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4777.674937] LustreError: 159420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4779.488935] LustreError: 159437:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4779.489663] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4779.537947] Lustre: Skipped 1 previous similar message [ 4779.545509] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4784.154466] Lustre: server umount lustre-MDT0001 complete [ 4794.159997] Lustre: server umount lustre-OST0000 complete [ 4803.735149] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 4813.531746] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 4848.064479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4848.257960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4848.294338] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4848.351327] Lustre: lustre-MDT0000: new disk, initializing [ 4848.415810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4848.432849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4852.389245] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4862.836450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4862.979671] Lustre: 166066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4863.020295] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4863.024513] Lustre: Skipped 1 previous similar message [ 4863.083292] Lustre: lustre-MDT0001: new disk, initializing [ 4863.164445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4863.213097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4863.228024] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4868.087645] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4878.896763] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4885.162532] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4892.657482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4892.961335] Lustre: lustre-OST0000: new disk, initializing [ 4892.966706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4892.979981] Lustre: 168036:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4893.068642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4898.876973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4898.894424] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4898.995711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4900.301844] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4909.383895] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4914.257860] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4914.431295] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4918.837862] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4919.037579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4924.900742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4924.921970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4924.947749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4924.956509] Lustre: Skipped 1 previous similar message [ 4935.136791] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4935.139509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4935.180208] Lustre: Skipped 3 previous similar messages [ 4935.537295] Lustre: server umount lustre-OST0000 complete [ 4945.376456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4945.382883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4945.387015] LustreError: Skipped 1 previous similar message [ 4945.394875] Lustre: Skipped 1 previous similar message [ 4946.886898] Lustre: server umount lustre-MDT0000 complete [ 4950.501756] LustreError: 166079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4950.522719] LustreError: 166079:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4951.209623] LustreError: 166059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789070801 with bad export cookie 5521385910175919814 [ 4951.214993] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4951.222184] LustreError: 166059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4951.736721] Lustre: server umount lustre-MDT0001 complete [ 4960.450788] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:06:48 (1789070808) [ 4978.919950] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 4989.248976] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 5026.169993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5026.434123] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5026.480339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5026.551990] Lustre: lustre-MDT0000: new disk, initializing [ 5026.635498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5026.655816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5031.444235] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5042.096564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5042.200640] Lustre: 172794:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5042.567452] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5047.256234] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5056.965419] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5063.927785] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5071.705916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5072.123892] Lustre: lustre-OST0000: new disk, initializing [ 5072.129772] Lustre: Skipped 1 previous similar message [ 5072.135215] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5072.143571] Lustre: Skipped 2 previous similar messages [ 5072.151631] Lustre: 174772:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5078.088153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5078.098137] Lustre: Skipped 1 previous similar message [ 5078.108550] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5078.200706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5079.130326] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5088.294965] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5098.983197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5098.987568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5099.003169] Lustre: Skipped 1 previous similar message [ 5099.008471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5099.031743] Lustre: Skipped 3 previous similar messages [ 5102.518739] Lustre: server umount lustre-OST0000 complete [ 5109.222317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5109.254224] Lustre: Skipped 2 previous similar messages [ 5113.101959] Lustre: server umount lustre-MDT0000 complete [ 5114.337337] LustreError: 172807:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5114.367146] LustreError: 172807:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5117.192883] LustreError: 172788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789070967 with bad export cookie 5521385910175920696 [ 5117.195299] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5117.203677] LustreError: 172788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5117.654564] Lustre: server umount lustre-MDT0001 complete [ 5128.336831] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:09:36 (1789070976) [ 5146.607906] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 5157.917807] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 5196.256183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5196.651805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5196.688800] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5196.818951] Lustre: lustre-MDT0000: new disk, initializing [ 5196.918898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5196.932336] Lustre: Skipped 2 previous similar messages [ 5196.955334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5201.807468] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5212.215035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5212.393438] Lustre: 179097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5212.668645] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5217.812709] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5227.018596] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5233.170308] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5240.168115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5240.423921] Lustre: 181073:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5246.521507] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5246.636454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5246.736708] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5256.390800] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5267.431025] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5267.433397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5267.439700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5267.439708] Lustre: Skipped 2 previous similar messages [ 5267.440326] LustreError: Skipped 1 previous similar message [ 5267.477559] Lustre: Skipped 1 previous similar message [ 5270.798666] Lustre: server umount lustre-OST0000 complete [ 5282.107933] Lustre: server umount lustre-MDT0000 complete [ 5282.794255] LustreError: 179106:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5282.818341] LustreError: 179106:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5285.852549] LustreError: 179091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789071135 with bad export cookie 5521385910175921781 [ 5285.853809] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5285.862801] LustreError: 179091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5286.251911] Lustre: server umount lustre-MDT0001 complete [ 5297.592943] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 5300.841893] Key type lgssc unregistered [ 5301.165489] LNet: 182930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5301.179665] LNetError: 182930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5301.197051] LNet: Removed LNI 192.168.203.137@tcp [ 5302.188244] Key type .llcrypt unregistered [ 5302.191298] Key type ._llcrypt unregistered [ 5331.465107] Key type ._llcrypt registered [ 5331.468744] Key type .llcrypt registered [ 5331.762832] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 5333.414670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5333.463537] alg: No test for adler32 (adler32-zlib) [ 5334.695679] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 5334.894704] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 5336.591254] Key type lgssc registered [ 5338.675934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5348.449913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5348.488277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5350.321772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5357.097612] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5360.612987] LustreError: 184344:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5360.656248] LustreError: 184344:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5365.733877] LustreError: 184343:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5368.157860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5368.835570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5375.035060] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5383.347075] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5390.644872] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5399.048369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5399.509789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5407.409556] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5416.928491] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5421.059097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5441.507632] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5441.535321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5441.551707] Lustre: Skipped 1 previous similar message [ 5446.431442] Lustre: server umount lustre-OST0000 complete [ 5450.729137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5450.738086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5450.755789] Lustre: Skipped 1 previous similar message [ 5450.768913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5451.749721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5456.761498] Lustre: server umount lustre-MDT0000 complete [ 5456.867966] LustreError: 184343:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5456.881961] LustreError: 184343:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5461.171424] LustreError: 184325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789071311 with bad export cookie 6983765491558618174 [ 5461.191555] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5461.715841] Lustre: server umount lustre-MDT0001 complete [ 5472.522401] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 5475.627890] Key type lgssc unregistered [ 5476.038694] LNet: 187902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5476.047866] LNetError: 187902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5476.076373] LNet: Removed LNI 192.168.203.137@tcp [ 5477.089623] Key type .llcrypt unregistered [ 5477.092145] Key type ._llcrypt unregistered [ 5504.117517] Key type ._llcrypt registered [ 5504.118920] Key type .llcrypt registered [ 5504.234515] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 5505.621652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5505.833464] alg: No test for adler32 (adler32-zlib) [ 5507.366895] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 5507.731709] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 5509.561568] Key type lgssc registered [ 5511.271450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5521.687891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5521.733474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5523.429542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5528.935435] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5533.670592] LustreError: 189316:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5533.687630] LustreError: 189316:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5538.430284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5539.003694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5544.208208] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5552.684381] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5559.029485] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5567.370168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5567.659821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5568.707023] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5575.356569] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5586.323194] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5593.176198] Lustre: lustre-OST0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 5594.598718] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5594.602326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5594.640381] LustreError: Skipped 1 previous similar message [ 5594.655361] Lustre: Skipped 1 previous similar message [ 5594.666413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5598.276217] Lustre: lustre-OST0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 5598.292949] Lustre: Skipped 1 previous similar message [ 5598.876467] Lustre: server umount lustre-OST0000 complete [ 5608.480453] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff88c79842f000.ost_server_uuid 50 [ 5610.710489] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5625.314587] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5625.319287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5625.327791] Lustre: Skipped 1 previous similar message [ 5625.349820] Lustre: Skipped 1 previous similar message [ 5630.438176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5630.454360] Lustre: Skipped 1 previous similar message [ 5631.342259] Lustre: server umount lustre-MDT0000 complete [ 5634.533363] LustreError: 189316:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5634.561195] LustreError: 189316:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5636.076716] LustreError: 189297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789071485 with bad export cookie 4018527865834844178 [ 5636.082056] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5636.084820] LustreError: 189297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5636.438424] Lustre: server umount lustre-MDT0001 complete [ 5650.148440] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 16:18:17 (1789071497) [ 5659.059718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5659.769420] LustreError: 193234:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5659.994900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5665.256431] LustreError: 193235:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5666.667285] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5669.346503] LustreError: 193234:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5677.590105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5678.065268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5683.801595] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5690.918398] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5697.218697] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5705.467152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5705.942381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5712.837993] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5721.121536] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5729.476676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5729.763656] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5729.907458] Lustre: lustre-OST0001: new disk, initializing [ 5729.911378] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5729.916860] Lustre: 196077:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5729.996825] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5737.563075] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5739.622748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5739.649373] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5739.731493] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5748.465633] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5753.312700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5753.329376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5753.345165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5755.368608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5755.376178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5757.208832] Lustre: server umount lustre-OST0000 complete [ 5759.532430] LustreError: 195056:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5759.562681] LustreError: 195056:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5763.869411] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5764.051717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5768.966442] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5769.148178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5775.844067] LustreError: 195056:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5775.865423] LustreError: 195056:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 5786.086490] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5786.090555] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5786.105758] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5786.105767] Lustre: Skipped 3 previous similar messages [ 5790.352847] Lustre: server umount lustre-OST0001 complete [ 5795.822856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5795.856898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5795.901530] Lustre: Skipped 1 previous similar message [ 5801.362494] Lustre: server umount lustre-MDT0000 complete [ 5805.110985] LustreError: 193216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789071654 with bad export cookie 4018527865834845249 [ 5805.131195] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5805.739500] Lustre: server umount lustre-MDT0001 complete [ 5843.076948] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 16:21:30 (1789071690) [ 5851.503769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5852.167649] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5852.237792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5852.620913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5858.135559] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5867.875448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5868.057627] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5868.117610] Lustre: 199517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5874.251929] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5883.310911] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5891.237088] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5901.010595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5901.484115] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5908.719729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5911.009096] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5919.786165] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5926.772436] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5926.855286] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5927.061552] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5927.065919] Lustre: Skipped 2 previous similar messages [ 5933.809591] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5934.338609] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5942.489727] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5948.511362] Lustre: lustre-OST0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 5948.515207] Lustre: Skipped 4 previous similar messages [ 5949.435630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5949.443898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5949.479747] Lustre: Skipped 1 previous similar message [ 5960.159145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5960.344762] Lustre: server umount lustre-OST0000 complete [ 5963.853073] LustreError: 201370:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5963.886501] LustreError: 201370:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 5980.138550] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5980.145710] LustreError: Skipped 1 previous similar message [ 5982.673433] Lustre: server umount lustre-OST0001 complete [ 5986.275187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5986.300377] Lustre: Skipped 3 previous similar messages [ 5992.256510] Lustre: server umount lustre-MDT0000 complete [ 5995.882794] LustreError: 199509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789071845 with bad export cookie 4018527865834846656 [ 5995.889957] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5995.894022] LustreError: 199509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5996.412744] Lustre: server umount lustre-MDT0001 complete [ 6026.646237] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 16:24:34 (1789071874) [ 6046.289965] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 6056.172564] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 6094.970158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6095.574034] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6095.638815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6095.773883] Lustre: lustre-MDT0000: new disk, initializing [ 6095.908703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6095.935987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6101.856538] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6113.449536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6113.625850] Lustre: 207623:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6113.716277] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6113.724949] Lustre: Skipped 1 previous similar message [ 6113.879796] Lustre: lustre-MDT0001: new disk, initializing [ 6114.109990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6114.120069] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6119.345410] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6130.033638] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6137.300542] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6145.913229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6146.192742] Lustre: lustre-OST0000: new disk, initializing [ 6146.197855] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6146.222827] Lustre: 209592:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6154.756742] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6156.451931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6156.475730] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6156.629751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6164.884507] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6169.960440] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6170.127203] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6174.415673] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6174.624503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6177.254589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6177.258372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6177.265206] LustreError: Skipped 2 previous similar messages [ 6177.273140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6177.279596] Lustre: Skipped 2 previous similar messages [ 6177.284423] Lustre: Skipped 13 previous similar messages [ 6182.727481] Lustre: server umount lustre-OST0000 complete [ 6192.609780] LustreError: 207634:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6192.633360] LustreError: 207634:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 6192.881968] Lustre: server umount lustre-MDT0000 complete [ 6196.306025] LustreError: 207614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789072046 with bad export cookie 4018527865834848448 [ 6196.314673] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6196.325750] LustreError: 207614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6196.791431] Lustre: server umount lustre-MDT0001 complete [ 6204.843573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6210.717632] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6222.460915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6230.000642] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6242.113635] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6251.875982] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6261.787777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6269.506997] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6277.859430] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6282.871249] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6283.138586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6288.126805] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6288.377740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6292.460104] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6296.607466] Lustre: server umount lustre-OST0000 complete [ 6305.001385] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6309.955837] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6341.957584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6348.705183] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6356.056294] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6357.700640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6357.717039] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6357.718692] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6357.744389] Lustre: Skipped 1 previous similar message [ 6357.760628] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6363.113438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6363.129356] Lustre: Skipped 4 previous similar messages [ 6363.138022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6363.144901] Lustre: Skipped 8 previous similar messages [ 6367.611834] Lustre: server umount lustre-OST0000 complete [ 6378.475001] LustreError: 214617:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6378.516422] LustreError: 214617:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6378.727240] Lustre: server umount lustre-MDT0000 complete [ 6384.623615] LustreError: 211786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789072234 with bad export cookie 4018527865834849330 [ 6384.643385] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6384.649854] LustreError: 211786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6385.806735] Lustre: server umount lustre-MDT0001 complete [ 6401.068648] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 16:30:47 (1789072247) [ 6408.294609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6408.759939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6408.769389] Lustre: Skipped 6 previous similar messages [ 6413.145571] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6421.763358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6426.875905] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6434.580751] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6441.087515] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6447.751857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6454.825936] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6462.748832] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6467.705704] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6467.900376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6473.344790] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6473.678720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6481.374124] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6481.657783] Lustre: lustre-OST0001: new disk, initializing [ 6481.669279] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6481.696710] Lustre: 221145:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6489.025609] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6489.220300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6489.255455] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6489.370486] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6499.840059] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6504.963664] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6505.205095] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6511.144921] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6511.349184] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6515.174875] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6515.209945] LustreError: Skipped 4 previous similar messages [ 6519.621438] Lustre: server umount lustre-OST0001 complete [ 6527.190921] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6527.524611] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6533.502654] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6533.884434] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6565.717755] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6571.508172] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6571.538820] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6571.544267] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6571.558009] Lustre: Skipped 1 previous similar message [ 6576.180608] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6586.552610] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6588.513625] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6605.279405] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6605.347192] Lustre: server umount lustre-OST0001 complete [ 6625.766821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6625.804726] Lustre: Skipped 7 previous similar messages [ 6625.841884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6625.871284] Lustre: Skipped 11 previous similar messages [ 6631.018260] Lustre: server umount lustre-OST0000 complete [ 6642.952522] LustreError: 224720:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x5:0x0].0x0 (0000000079ce1a6a) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6643.057214] Lustre: server umount lustre-MDT0000 complete [ 6643.168914] LustreError: 217777:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6643.215569] LustreError: 217777:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 41 previous similar messages [ 6648.191121] LustreError: 217757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789072498 with bad export cookie 4018527865834850296 [ 6648.201544] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6648.207869] LustreError: 217757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6654.722929] Lustre: server umount lustre-MDT0001 complete [ 6689.209553] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 16:35:37 (1789072537) [ 6701.013735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6701.246989] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6701.267667] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6706.914593] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6716.436210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6716.525843] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6716.553941] Lustre: 226394:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6722.387584] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6731.957781] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6738.448207] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6746.917175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6747.161600] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6756.032312] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6764.429353] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6774.563244] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6774.614656] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6781.833821] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6791.388528] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6793.807723] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6799.808389] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6799.973890] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6805.170127] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6805.423683] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6811.898503] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88c782ee4000.ost_server_uuid 50 [ 6813.846570] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88c782ee4000.ost_server_uuid in IDLE state after 0 sec [ 6814.996973] Lustre: Permanently deactivating lustre-OST0001 [ 6815.037581] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6820.305282] Lustre: Permanently reactivating lustre-OST0001 [ 6820.354735] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6820.365660] Lustre: Skipped 2 previous similar messages [ 6828.518166] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6828.530594] LustreError: Skipped 5 previous similar messages [ 6832.454946] Lustre: server umount lustre-OST0001 complete [ 6843.692762] Lustre: server umount lustre-OST0000 complete [ 6864.354032] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6864.686392] Lustre: server umount lustre-MDT0000 complete [ 6869.579119] LustreError: 229268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789072719 with bad export cookie 4018527865834851584 [ 6869.581499] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6869.588873] LustreError: 229268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6869.892913] Lustre: server umount lustre-MDT0001 complete [ 6899.022182] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 16:39:06 (1789072746) [ 6911.131151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6912.127162] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6912.238204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6912.262650] Lustre: Skipped 2 previous similar messages [ 6919.970258] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6932.832616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6932.909399] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6932.951500] Lustre: 233097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6933.126213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6933.130362] Lustre: Skipped 9 previous similar messages [ 6938.887972] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6948.617314] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6955.036524] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6961.992824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6962.201210] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6962.231166] Lustre: Permanently deactivating lustre-OST0000 [ 6962.239447] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6969.827660] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6972.465706] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6972.475076] LustreError: 235440:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 6972.488198] LustreError: 235440:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 6977.581979] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6985.391744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6985.498864] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6992.271512] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6993.846050] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7006.074266] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7008.676500] Lustre: Permanently reactivating lustre-OST0000 [ 7008.692948] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7008.702558] Lustre: Skipped 2 previous similar messages [ 7015.519762] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7015.520082] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 43 seconds. I think it's dead, and I am evicting it. exp 000000001152197a, cur 1789072865 deadline 1789072852 last 1789072822 [ 7015.539321] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7015.585415] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7015.658042] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7028.900867] Lustre: server umount lustre-OST0001 complete [ 7041.311267] Lustre: server umount lustre-OST0000 complete [ 7055.765139] Lustre: server umount lustre-MDT0000 complete [ 7062.249594] LustreError: 233842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789072912 with bad export cookie 4018527865834853257 [ 7062.261905] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7062.297086] LustreError: 233842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7063.575981] Lustre: server umount lustre-MDT0001 complete [ 7076.531044] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 7079.511594] Key type lgssc unregistered [ 7079.937512] LNet: 237980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7079.963333] LNetError: 237980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7079.992772] LNet: Removed LNI 192.168.203.137@tcp [ 7081.358273] Key type .llcrypt unregistered [ 7081.361043] Key type ._llcrypt unregistered [ 7117.657828] Key type ._llcrypt registered [ 7117.659505] Key type .llcrypt registered [ 7117.936797] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 7119.006985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7119.087473] alg: No test for adler32 (adler32-zlib) [ 7120.792843] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 7121.091093] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 7122.935257] Key type lgssc registered [ 7124.792747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7136.205282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7136.253714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7137.701827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7142.690559] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7148.002382] LustreError: 239444:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7148.029343] LustreError: 239444:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7153.120243] LustreError: 239443:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7153.280663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7153.427662] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7153.465185] Lustre: 239433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7153.475493] Lustre: Permanently deactivating lustre-MDT0001 [ 7153.493903] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7153.576079] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7153.585829] LustreError: 239474:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7153.598470] LustreError: 239474:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7153.607939] LustreError: 239474:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7153.968949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7158.589043] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7167.226110] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7173.733425] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7181.841728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7182.264138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7183.277058] LustreError: 241286:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7183.293990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7188.463034] LustreError: 241287:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7190.157224] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7193.571611] LustreError: 241286:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7198.462247] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7198.711510] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 7203.828963] LustreError: 241287:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7203.850746] LustreError: 241287:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7205.583640] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7205.943395] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7211.032335] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7211.033550] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7214.214051] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7224.470611] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7228.140146] Lustre: Permanently reactivating lustre-MDT0001 [ 7228.162442] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7228.170706] Lustre: Skipped 2 previous similar messages [ 7234.656971] Lustre: lustre-MDT0001: Client ec3a5188-99d5-4ce5-a9e6-04321eeee955 (at 192.168.203.37@tcp) reconnecting [ 7241.320435] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7242.714745] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7250.205242] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7250.485932] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7256.142313] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7256.348481] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7259.239738] Lustre: Permanently deactivating lustre-MDT0001 [ 7259.262582] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7259.268895] Lustre: Skipped 2 previous similar messages [ 7269.604987] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7282.147716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7282.158292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7282.188329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7282.666461] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7282.684298] Lustre: Skipped 2 previous similar messages [ 7282.692433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7282.699987] Lustre: Skipped 2 previous similar messages [ 7287.784286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7287.804408] Lustre: Skipped 3 previous similar messages [ 7289.939488] Lustre: server umount lustre-MDT0000 complete [ 7292.919450] LustreError: 240472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7292.965439] LustreError: 240472:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7295.222579] LustreError: 239423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789073145 with bad export cookie 8668737341439895566 [ 7295.224545] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7295.231361] LustreError: 239423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7295.566505] Lustre: server umount lustre-MDT0001 complete [ 7302.900224] Lustre: server umount lustre-OST0000 complete [ 7316.632896] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 16:46:04 (1789073164) [ 7333.175882] Lustre: server umount lustre-OST0001 complete [ 7341.984778] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 7355.705444] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 7389.810405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7390.085324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7390.098435] Lustre: Skipped 2 previous similar messages [ 7390.149384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7390.239713] Lustre: lustre-MDT0000: new disk, initializing [ 7390.331618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7390.363538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7394.821781] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7404.772273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7404.886451] Lustre: 247761:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7404.958149] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7404.964401] Lustre: Skipped 1 previous similar message [ 7405.075590] Lustre: lustre-MDT0001: new disk, initializing [ 7405.196803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7405.245658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7405.263508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7409.449767] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7418.300976] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7426.647598] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7434.880370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7435.211806] Lustre: lustre-OST0000: new disk, initializing [ 7435.215198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7435.222246] Lustre: 249729:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7435.326628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7441.673248] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7444.150099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7444.173132] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7444.306630] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7454.215170] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7458.797443] LustreError: 250310:0:(mdt_handler.c:3332:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7468.803655] LustreError: 250310:0:(mdt_handler.c:3332:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7468.900727] LustreError: 247769:0:(mdt_handler.c:3332:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7472.345638] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7472.671746] Lustre: lustre-OST0001: new disk, initializing [ 7472.698834] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7472.748352] Lustre: 250847:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7472.757312] Lustre: lustre-OST0001: Not available for connect from 192.168.203.37@tcp (not set up) [ 7473.092281] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7478.718661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7478.725286] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7478.888261] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7478.985599] LustreError: 247769:0:(mdt_handler.c:3332:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7482.825128] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7494.055098] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7495.910967] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7499.252269] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7499.260833] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7499.261791] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7499.279270] Lustre: Skipped 1 previous similar message [ 7499.316943] Lustre: Skipped 1 previous similar message [ 7504.354793] LustreError: 249715:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7504.465726] Lustre: server umount lustre-OST0001 complete [ 7524.839147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7524.868235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7524.873202] Lustre: Skipped 1 previous similar message [ 7524.892147] Lustre: Skipped 1 previous similar message [ 7529.163977] Lustre: server umount lustre-OST0000 complete [ 7534.048306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7534.057602] LustreError: Skipped 1 previous similar message [ 7534.063062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7539.803338] Lustre: server umount lustre-MDT0000 complete [ 7544.313838] LustreError: 247755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789073394 with bad export cookie 8668737341439898604 [ 7544.316570] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7544.330698] LustreError: 247755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7550.914248] Lustre: server umount lustre-MDT0001 complete [ 7563.247817] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 7566.751282] Key type lgssc unregistered [ 7567.027921] LNet: 252881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7567.042793] LNetError: 252881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7567.094757] LNet: Removed LNI 192.168.203.137@tcp [ 7568.252412] Key type .llcrypt unregistered [ 7568.262664] Key type ._llcrypt unregistered [ 7619.883296] Key type ._llcrypt registered [ 7619.885348] Key type .llcrypt registered [ 7620.065909] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 7621.520830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7621.560277] alg: No test for adler32 (adler32-zlib) [ 7622.661193] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 7622.962164] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 7624.655360] Key type lgssc registered [ 7626.225642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7634.764580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7634.812471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7636.270760] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7636.294579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7636.578872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7641.891817] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7653.117240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7653.223137] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7653.249609] Lustre: 255088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7653.608789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7658.388117] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7666.362871] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7673.417413] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7681.296970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7681.419415] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7681.546287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7687.397953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7689.973646] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7699.219443] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7753.696782] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7753.713445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7753.736556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7754.222126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7754.239117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7754.254752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7754.265212] Lustre: Skipped 1 previous similar message [ 7758.394697] Lustre: server umount lustre-OST0000 complete [ 7763.114460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7777.234680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7777.605272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7779.635160] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7779.663142] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7783.945896] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7791.047726] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7795.058754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7795.082330] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7836.130891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7836.140174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7836.142056] LustreError: Skipped 1 previous similar message [ 7836.143310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7836.185406] Lustre: Skipped 1 previous similar message [ 7837.819768] Lustre: server umount lustre-OST0000 complete [ 7843.298163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7843.316152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7843.339904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7843.353621] Lustre: Skipped 1 previous similar message [ 7848.897437] Lustre: server umount lustre-MDT0000 complete [ 7854.227441] LustreError: 255080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789073704 with bad export cookie 5771636651190843860 [ 7854.229210] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7854.245692] LustreError: 255080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7854.495227] Lustre: server umount lustre-MDT0001 complete [ 7867.349405] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 7870.486233] Key type lgssc unregistered [ 7870.832812] LNet: 260510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7870.845547] LNetError: 260510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7870.878000] LNet: Removed LNI 192.168.203.137@tcp [ 7871.959953] Key type .llcrypt unregistered [ 7871.964966] Key type ._llcrypt unregistered [ 7899.149363] Key type ._llcrypt registered [ 7899.152643] Key type .llcrypt registered [ 7899.290884] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 7900.488502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7900.529411] alg: No test for adler32 (adler32-zlib) [ 7901.685785] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 7901.970247] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 7903.647750] Key type lgssc registered [ 7904.916981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7915.062304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7915.099782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7916.667427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7922.305979] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7926.248165] LustreError: 261926:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7926.278508] LustreError: 261926:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7931.373713] LustreError: 261925:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7933.635541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7934.242223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7940.589810] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7951.463764] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7959.254430] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7968.553309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7969.126371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7977.447412] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7986.469184] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7986.690212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8024.552785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8024.565363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8024.606112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8025.850618] Lustre: server umount lustre-OST0000 complete [ 8031.717139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8031.726669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8031.773058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8032.743804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8037.864921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8037.873854] Lustre: Skipped 3 previous similar messages [ 8041.025406] Lustre: server umount lustre-MDT0000 complete [ 8042.978568] LustreError: 261925:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8042.993756] LustreError: 261925:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8045.652619] LustreError: 261906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789073895 with bad export cookie 18298218465548810921 [ 8045.681563] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8046.302600] Lustre: server umount lustre-MDT0001 complete [ 8063.296779] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 8067.013948] Key type lgssc unregistered [ 8067.236756] LNet: 266234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8067.244435] LNetError: 266234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8067.260361] LNet: Removed LNI 192.168.203.137@tcp [ 8068.279421] Key type .llcrypt unregistered [ 8068.281852] Key type ._llcrypt unregistered [ 8089.187050] Key type ._llcrypt registered [ 8089.190732] Key type .llcrypt registered [ 8089.404621] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 8090.618449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8090.726537] alg: No test for adler32 (adler32-zlib) [ 8092.221374] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 8092.808968] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 8094.664388] Key type lgssc registered [ 8096.481991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8104.612985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8104.643578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8106.133928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8111.187268] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8115.681690] LustreError: 267452:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8115.705131] LustreError: 267452:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8119.790984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8120.046122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8124.427299] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8133.861810] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8141.445530] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8152.106139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8152.623489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8160.259176] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8168.177996] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8175.222741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8190.946661] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8190.961568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8190.985573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8190.988755] Lustre: Skipped 1 previous similar message [ 8196.066077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8196.077308] Lustre: Skipped 2 previous similar messages [ 8196.829870] Lustre: server umount lustre-OST0000 complete [ 8201.188865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8201.195114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8201.231708] Lustre: Skipped 1 previous similar message [ 8201.253311] Lustre: Skipped 1 previous similar message [ 8206.306812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8206.319778] Lustre: Skipped 1 previous similar message [ 8207.472864] Lustre: server umount lustre-MDT0000 complete [ 8211.429031] LustreError: 268180:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8211.439622] LustreError: 268180:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8212.756888] LustreError: 267433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789074062 with bad export cookie 9513499596573688794 [ 8212.764683] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8212.770133] LustreError: 267433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8213.116675] Lustre: server umount lustre-MDT0001 complete [ 8225.935836] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 8229.100549] Key type lgssc unregistered [ 8229.526032] LNet: 271167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8229.553694] LNetError: 271167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8229.581064] LNet: Removed LNI 192.168.203.137@tcp [ 8230.562809] Key type .llcrypt unregistered [ 8230.566268] Key type ._llcrypt unregistered [ 8259.635973] Key type ._llcrypt registered [ 8259.640467] Key type .llcrypt registered [ 8259.803624] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 8261.253200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8261.416912] alg: No test for adler32 (adler32-zlib) [ 8262.627690] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 8262.923722] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 8264.783260] Key type lgssc registered [ 8266.070165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8276.161364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8276.209363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8277.753453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8282.824531] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8287.212972] LustreError: 272587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8287.229227] LustreError: 272587:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8292.329644] LustreError: 272588:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8293.225567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8294.022698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8300.251113] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8311.921684] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8319.233510] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8327.174355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8327.677256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8335.666376] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8343.680535] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8348.161082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8379.874048] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8379.892104] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8379.915657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8383.972624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8383.981163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8383.989062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8385.294266] Lustre: server umount lustre-OST0000 complete [ 8391.151358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8391.167076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8391.208767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8391.224916] Lustre: Skipped 1 previous similar message [ 8394.222164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8394.224640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8396.899753] Lustre: server umount lustre-MDT0000 complete [ 8399.330241] LustreError: 272587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8399.362266] LustreError: 272587:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8400.491081] LustreError: 272568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789074250 with bad export cookie 9967209872245963993 [ 8400.509230] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8400.518186] LustreError: 272568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8400.849602] Lustre: server umount lustre-MDT0001 complete [ 8412.355415] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 8415.532373] Key type lgssc unregistered [ 8415.950969] LNet: 276895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8415.971553] LNetError: 276895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8416.997859] LNet: Removed LNI 192.168.203.137@tcp [ 8418.102545] Key type .llcrypt unregistered [ 8418.104960] Key type ._llcrypt unregistered [ 8436.254565] Key type ._llcrypt registered [ 8436.256160] Key type .llcrypt registered [ 8436.379572] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 8437.565431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8437.667843] alg: No test for adler32 (adler32-zlib) [ 8438.762599] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 8439.001091] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 8440.687269] Key type lgssc registered [ 8441.634195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8449.307130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8449.329374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8450.781034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8454.474038] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8460.260358] LustreError: 278128:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8460.276444] LustreError: 278128:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8462.753236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8463.145437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8467.905985] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8476.163945] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8484.078693] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8493.548975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8493.951288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8496.001431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8501.592739] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8511.193981] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8526.817339] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8526.827496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8526.861052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8527.843511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8527.847749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8527.881498] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8527.890742] Lustre: Skipped 1 previous similar message [ 8530.573594] Lustre: server umount lustre-OST0000 complete [ 8535.012086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8535.022833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8535.037658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8538.083118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8538.083145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8542.663763] Lustre: server umount lustre-MDT0000 complete [ 8543.201505] LustreError: 278132:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8543.213236] LustreError: 278132:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8545.866997] LustreError: 278095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789074395 with bad export cookie 16232690292819356815 [ 8545.881080] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8546.083475] Lustre: server umount lustre-MDT0001 complete [ 8556.367182] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 8558.808317] Key type lgssc unregistered [ 8559.081780] LNet: 281828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8559.087848] LNetError: 281828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8559.104086] LNet: Removed LNI 192.168.203.137@tcp [ 8559.868108] Key type .llcrypt unregistered [ 8559.869930] Key type ._llcrypt unregistered [ 8570.239126] Key type ._llcrypt registered [ 8570.244434] Key type .llcrypt registered [ 8570.351479] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8586.761310] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 8603.308963] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 8604.383482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8604.404070] alg: No test for adler32 (adler32-zlib) [ 8605.575173] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 8605.884443] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 8607.751718] Key type lgssc registered [ 8609.374596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8649.836476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8649.855262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8651.020977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8651.057449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8651.140580] Lustre: lustre-MDT0000: new disk, initializing [ 8651.216524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8651.232818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8655.375612] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8665.068510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8665.216332] Lustre: 285387:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8665.272836] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8665.277740] Lustre: Skipped 1 previous similar message [ 8665.338465] Lustre: lustre-MDT0001: new disk, initializing [ 8665.469545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8665.520327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8665.537638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8670.227136] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8678.310884] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8683.185729] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8689.090430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8689.367349] Lustre: lustre-OST0000: new disk, initializing [ 8689.371517] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8689.377658] Lustre: 287357:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8689.485138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8695.092159] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8696.373776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8696.382346] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8696.520287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8703.093806] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8707.137956] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8707.329279] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8711.629914] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8711.818988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8717.285295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8717.298539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8717.309660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8717.323330] Lustre: Skipped 1 previous similar message [ 8719.740597] Lustre: server umount lustre-OST0000 complete [ 8727.016101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8727.024664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8727.044037] Lustre: Skipped 1 previous similar message [ 8727.049429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8732.640552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8732.647521] Lustre: Skipped 2 previous similar messages [ 8737.247167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8737.479307] Lustre: server umount lustre-MDT0000 complete [ 8741.495401] LustreError: 287356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789074591 with bad export cookie 9528908417360886028 [ 8741.513373] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8741.518312] LustreError: 287356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8741.808284] Lustre: server umount lustre-MDT0001 complete [ 8752.110806] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 17:10:00 (1789074600) [ 8761.198634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8761.679034] LustreError: 289771:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8761.692028] LustreError: 289771:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8761.734855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8766.622174] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8766.950246] LustreError: 289772:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8772.066370] LustreError: 289771:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8774.536931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8774.884464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8780.347141] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8789.812788] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8795.511846] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8802.116258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8802.374699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8807.923519] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8815.146726] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8830.436035] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8830.442145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8830.451325] Lustre: Skipped 2 previous similar messages [ 8834.648657] Lustre: server umount lustre-OST0000 complete [ 8840.681557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8840.690166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8840.707103] Lustre: Skipped 1 previous similar message [ 8840.722109] Lustre: Skipped 2 previous similar messages [ 8844.511946] Lustre: server umount lustre-MDT0000 complete [ 8845.796888] LustreError: 289767:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8845.825515] LustreError: 289767:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8848.062144] LustreError: 289752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789074697 with bad export cookie 9528908417360886910 [ 8848.080326] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8848.094307] LustreError: 289752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8848.463057] Lustre: server umount lustre-MDT0001 complete [ 8859.156328] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 8862.082368] Key type lgssc unregistered [ 8862.391032] LNet: 293324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8862.412866] LNetError: 293324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8862.435748] LNet: Removed LNI 192.168.203.137@tcp [ 8863.235340] Key type .llcrypt unregistered [ 8863.238524] Key type ._llcrypt unregistered [ 8888.502273] Key type ._llcrypt registered [ 8888.504527] Key type .llcrypt registered [ 8888.599597] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 8904.066473] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 8905.625774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8905.746920] alg: No test for adler32 (adler32-zlib) [ 8906.868922] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 8907.127483] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 8908.823194] Key type lgssc registered [ 8910.121518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8952.762886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8952.796175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8954.030604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8954.090857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8954.197331] Lustre: lustre-MDT0000: new disk, initializing [ 8954.259854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8954.276396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8958.440667] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8969.437430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8969.627672] Lustre: 297148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8969.730681] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8969.743890] Lustre: Skipped 1 previous similar message [ 8969.899711] Lustre: lustre-MDT0001: new disk, initializing [ 8970.004759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8970.069639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8970.076026] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8976.355482] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 8989.854605] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8996.935184] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9006.084197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9006.415484] Lustre: lustre-OST2710: new disk, initializing [ 9006.424992] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9006.433759] Lustre: 299118:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9006.535609] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9013.337229] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9013.390293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9013.408324] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9013.516083] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9022.363742] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9029.995372] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9030.207291] Lustre: lustre-OST03e8: new disk, initializing [ 9030.210831] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9030.220237] Lustre: 300182:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9030.313752] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9035.876877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9035.889097] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9035.993614] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9036.457145] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9042.898741] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9050.166367] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9050.335487] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9055.761460] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9056.027918] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9061.708316] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9061.999580] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9066.446225] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9066.707866] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9072.097916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9072.101679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9072.118932] Lustre: Skipped 1 previous similar message [ 9072.134728] Lustre: Skipped 2 previous similar messages [ 9077.090845] Lustre: server umount lustre-MDT0000 complete [ 9080.570030] LustreError: 300181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789074930 with bad export cookie 10989611610613149215 [ 9080.572635] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9080.600766] LustreError: 300181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9080.962669] Lustre: server umount lustre-MDT0001 complete [ 9095.205368] Lustre: server umount lustre-OST2710 complete [ 9098.722357] Lustre: 294943:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789074932/real 1789074932] req@00000000df38b5c0 x1875980851416576/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789074948 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9098.749592] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9098.761468] Lustre: Skipped 2 previous similar messages [ 9099.000498] Lustre: server umount lustre-OST03e8 complete [ 9116.186214] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 9127.277379] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 9168.472304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9168.823683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9168.932883] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9169.055829] Lustre: lustre-MDT0000: new disk, initializing [ 9169.185372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9169.221741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9174.392626] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9185.149224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9185.298084] Lustre: 305427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9185.345829] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9185.350860] Lustre: Skipped 1 previous similar message [ 9185.465221] Lustre: lustre-MDT0001: new disk, initializing [ 9185.553238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9185.614374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9185.628573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9190.312882] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9200.713554] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9206.936496] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9215.993234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9216.371679] Lustre: lustre-OST0000: new disk, initializing [ 9216.381962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9216.395595] Lustre: 307399:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9216.533697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9224.321462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9224.334849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9224.349317] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9224.594233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9234.628322] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9240.146611] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9240.439740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9245.754700] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9246.034713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9250.282673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9250.289516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9250.305681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9250.309929] LustreError: Skipped 1 previous similar message [ 9250.335619] Lustre: Skipped 1 previous similar message [ 9254.206102] Lustre: server umount lustre-OST0000 complete [ 9260.513356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9260.522324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9260.547414] Lustre: Skipped 2 previous similar messages [ 9260.563076] Lustre: Skipped 1 previous similar message [ 9265.632891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9265.649373] Lustre: Skipped 1 previous similar message [ 9270.753474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9270.764860] Lustre: Skipped 1 previous similar message [ 9272.287146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9272.457654] Lustre: server umount lustre-MDT0000 complete [ 9275.876523] LustreError: 305440:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9275.898054] LustreError: 305440:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 9276.163739] LustreError: 305420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789075126 with bad export cookie 10989611610613150888 [ 9276.167239] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9276.178721] LustreError: 305420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9276.417589] Lustre: server umount lustre-MDT0001 complete [ 9285.422512] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 17:18:53 (1789075133) [ 9287.046503] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9289.057816] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 17:18:57 (1789075137) [ 9292.462211] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 9319.894741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9320.194893] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9320.241280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9320.429302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9325.010065] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9333.853493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9333.985882] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9334.038214] Lustre: 311261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9334.125375] LustreError: 311271:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9339.454987] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9361.453080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9361.691858] Lustre: Denying initial registration attempt from nid 192.168.203.137@tcp, specified as failover [ 9361.705180] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9361.715649] LustreError: 312714:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9361.723740] LustreError: 312714:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9361.730744] LustreError: 312714:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9361.810742] Lustre: server umount lustre-OST0000 complete [ 9361.816619] LustreError: 312714:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9370.081538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9370.137031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9370.148908] Lustre: Skipped 1 previous similar message [ 9375.727043] Lustre: server umount lustre-MDT0000 complete [ 9380.327206] LustreError: 311271:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9380.351737] LustreError: 311271:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9380.411808] LustreError: 311253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789075230 with bad export cookie 10989611610613151777 [ 9380.414164] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9380.424056] LustreError: 311253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9381.007241] Lustre: server umount lustre-MDT0001 complete [ 9393.208463] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 9396.209131] Key type lgssc unregistered [ 9396.590102] LNet: 313837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9396.604261] LNetError: 313837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9396.630835] LNet: Removed LNI 192.168.203.137@tcp [ 9397.609242] Key type .llcrypt unregistered [ 9397.611940] Key type ._llcrypt unregistered [ 9410.785592] Key type ._llcrypt registered [ 9410.794421] Key type .llcrypt registered [ 9410.986811] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 9412.276807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9412.351406] alg: No test for adler32 (adler32-zlib) [ 9413.421102] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 9413.644747] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 9415.295512] Key type lgssc registered [ 9416.394954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9469.471344] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 9481.203774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9481.278089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9482.661420] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9482.694697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9482.954799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9488.267953] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9498.129835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9498.356217] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9498.396034] Lustre: 316599:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9498.801482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9504.363934] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9513.007505] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9519.204950] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9539.417388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9539.675649] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9539.980754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9550.203730] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9563.006604] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9566.702258] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9566.711208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9566.712435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9566.752209] Lustre: Skipped 1 previous similar message [ 9572.041432] Lustre: server umount lustre-OST0000 complete [ 9581.030349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9581.049080] LustreError: Skipped 1 previous similar message [ 9581.055311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9581.091241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9581.108467] Lustre: Skipped 1 previous similar message [ 9583.791612] Lustre: server umount lustre-MDT0000 complete [ 9587.168031] LustreError: 316610:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9587.214439] LustreError: 316610:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9589.249213] LustreError: 316592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789075439 with bad export cookie 16193171166575712816 [ 9589.254532] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9589.269302] LustreError: 316592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9589.814433] Lustre: server umount lustre-MDT0001 complete [ 9605.189551] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 9609.289620] Key type lgssc unregistered [ 9609.671661] LNet: 320176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9609.685607] LNetError: 320176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9610.726935] LNet: Removed LNI 192.168.203.137@tcp [ 9612.163270] Key type .llcrypt unregistered [ 9612.173160] Key type ._llcrypt unregistered [ 9645.534708] Key type ._llcrypt registered [ 9645.549632] Key type .llcrypt registered [ 9645.742477] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 9647.363195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9647.486896] alg: No test for adler32 (adler32-zlib) [ 9648.645286] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 9648.923809] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 9650.647260] Key type lgssc registered [ 9652.510960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9661.151823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9661.179657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9662.818245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9668.652917] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9673.201423] LustreError: 321591:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9673.231944] LustreError: 321591:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9678.307567] LustreError: 321590:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9679.018768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9679.556624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9685.436925] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9695.015754] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9702.577444] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9712.750356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9713.266900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9720.311522] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9729.952554] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9744.356650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9744.361191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9744.370581] Lustre: Skipped 2 previous similar messages [ 9744.394232] Lustre: Skipped 2 previous similar messages [ 9746.000524] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 9749.476556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9754.593756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9754.604694] Lustre: Skipped 5 previous similar messages [ 9756.127155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9756.436550] Lustre: server umount lustre-MDT0000 complete [ 9759.712849] LustreError: 321591:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9759.757212] LustreError: 321591:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9760.546508] LustreError: 321570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789075610 with bad export cookie 2605358268091287453 [ 9760.564184] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9761.359531] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.37@tcp (stopping) [ 9761.375370] Lustre: Skipped 1 previous similar message [ 9764.832777] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9764.833535] LustreError: 323673:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9764.887896] LustreError: 323673:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9767.017799] Lustre: server umount lustre-MDT0001 complete [ 9775.390892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9786.051144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9806.257061] LustreError: 323423:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9806.295173] LustreError: 323423:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9806.366572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9806.427468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9811.449978] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9815.525278] LustreError: 325188:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9815.542841] LustreError: 325188:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9820.665729] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9821.371256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9821.882124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9827.242868] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9827.308587] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9836.234889] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9837.902513] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9844.860783] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9846.861611] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9857.505148] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9857.526134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9857.554243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9857.562368] Lustre: Skipped 3 previous similar messages [ 9858.017215] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9860.243430] Lustre: server umount lustre-OST0000 complete [ 9868.262567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9868.267349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9868.287647] Lustre: Skipped 2 previous similar messages [ 9871.272897] Lustre: server umount lustre-MDT0000 complete [ 9873.380448] LustreError: 325194:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9873.418844] LustreError: 325194:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 9876.209693] LustreError: 321570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789075726 with bad export cookie 2605358268091294621 [ 9876.215230] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9876.239575] LustreError: 321570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9876.932024] Lustre: server umount lustre-MDT0001 complete [ 9891.640414] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 9895.431757] Key type lgssc unregistered [ 9895.867800] LNet: 327867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9895.871939] LNetError: 327867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9895.897554] LNet: Removed LNI 192.168.203.137@tcp [ 9897.011686] Key type .llcrypt unregistered [ 9897.019839] Key type ._llcrypt unregistered [ 9927.939380] Key type ._llcrypt registered [ 9927.941403] Key type .llcrypt registered [ 9928.124991] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 9929.654736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9929.796519] alg: No test for adler32 (adler32-zlib) [ 9930.975368] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 9931.279831] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 9932.999224] Key type lgssc registered [ 9934.517919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9944.657462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9944.685791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9946.408841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9952.121594] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9956.850092] LustreError: 329283:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9956.876599] LustreError: 329283:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9961.970078] LustreError: 329282:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9963.445903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9963.916498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9970.112798] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 9979.476893] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9987.185453] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9990.922497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9990.927924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9990.946582] Lustre: Skipped 1 previous similar message [ 9990.977765] Lustre: Skipped 1 previous similar message [ 9996.259990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9996.275657] Lustre: Skipped 1 previous similar message [ 9997.387714] Lustre: server umount lustre-MDT0000 complete [10001.376800] LustreError: 330011:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10001.396497] LustreError: 330011:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10002.035457] LustreError: 329262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789075851 with bad export cookie 6987887776458126984 [10002.045080] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10002.063183] LustreError: 329262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10002.642260] Lustre: server umount lustre-MDT0001 complete [10024.105620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10024.497965] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10024.563614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10025.061221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10031.284209] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10040.281569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10040.357469] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10040.385589] Lustre: 331871:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10040.721056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10045.634342] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10054.654917] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10061.289545] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10072.828953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10073.368877] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10073.383759] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10073.400870] LustreError: 333757:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [10073.412716] LustreError: 333757:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10073.425219] LustreError: 333757:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10073.560434] Lustre: server umount lustre-OST0000 complete [10073.562735] LustreError: 333757:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10087.680550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10088.178030] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10088.638776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10095.374357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10097.333582] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10105.457348] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10113.774659] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10113.869096] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10113.946321] Lustre: lustre-OST0001: new disk, initializing [10113.954651] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10113.964633] Lustre: 335298:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10114.098584] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10121.670197] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10122.321101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10122.336908] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10122.468922] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10131.130430] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10138.080326] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10138.087432] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10138.104069] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10138.123883] Lustre: Skipped 1 previous similar message [10140.424649] Lustre: server umount lustre-OST0001 complete [10143.209160] LustreError: 334226:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10143.248438] LustreError: 334226:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10146.784470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10146.786106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10146.794463] LustreError: Skipped 1 previous similar message [10146.825374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10146.844258] Lustre: Skipped 2 previous similar messages [10148.323365] LustreError: 334224:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10148.349585] LustreError: 334224:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10151.640610] Lustre: server umount lustre-OST0000 complete [10158.562052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10158.563141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10158.575537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10158.592937] Lustre: Skipped 2 previous similar messages [10158.611926] Lustre: Skipped 3 previous similar messages [10162.157350] Lustre: server umount lustre-MDT0000 complete [10163.681195] LustreError: 333392:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10163.709959] LustreError: 333392:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10166.221241] LustreError: 331862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789076016 with bad export cookie 6987887776458127537 [10166.233916] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10166.239671] LustreError: 331862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10166.750201] Lustre: server umount lustre-MDT0001 complete [10179.181899] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [10183.013067] Key type lgssc unregistered [10183.270598] LNet: 337258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10183.278762] LNetError: 337258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10183.300374] LNet: Removed LNI 192.168.203.137@tcp [10184.209389] Key type .llcrypt unregistered [10184.211762] Key type ._llcrypt unregistered [10268.655497] Key type ._llcrypt registered [10268.662037] Key type .llcrypt registered [10268.864161] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [10285.730718] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [10287.970765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10288.006034] alg: No test for adler32 (adler32-zlib) [10289.118291] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [10289.498674] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [10291.263518] Key type lgssc registered [10293.197350] Lustre: Echo OBD driver; http://www.lustre.org/ [10338.458785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10338.494287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10339.797460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10339.824774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10339.971704] Lustre: lustre-MDT0000: new disk, initializing [10340.155089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10340.181438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10346.134821] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10362.130714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10362.255565] Lustre: 342195:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10362.341411] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10362.348962] Lustre: Skipped 1 previous similar message [10362.407634] Lustre: lustre-MDT0001: new disk, initializing [10362.569319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10362.658316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10362.682994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10370.795169] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10383.746597] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10390.379943] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10398.571053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10399.064288] Lustre: lustre-OST0000: new disk, initializing [10399.078097] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10399.091577] Lustre: 344165:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10399.266444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10406.397853] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10408.614421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10408.644610] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10408.769357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10416.966548] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10423.754671] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10424.092844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10431.212986] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10431.505962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10434.529612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10434.544544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10434.559035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10434.577170] Lustre: Skipped 1 previous similar message [10434.603062] Lustre: Skipped 1 previous similar message [10439.648619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10439.657120] Lustre: Skipped 1 previous similar message [10444.817123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10444.835217] Lustre: Skipped 1 previous similar message [10448.351105] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10448.847408] Lustre: server umount lustre-OST0000 complete [10455.008831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10455.014598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10455.020565] LustreError: Skipped 1 previous similar message [10455.027869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10455.079262] Lustre: Skipped 1 previous similar message [10460.129415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10460.132754] Lustre: Skipped 1 previous similar message [10460.863992] Lustre: server umount lustre-MDT0000 complete [10465.213471] LustreError: 342187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789076315 with bad export cookie 15528515119435249798 [10465.231240] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10465.243911] LustreError: 342187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10466.024897] Lustre: server umount lustre-MDT0001 complete [10481.555507] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 17:38:47 (1789076327) [10490.946551] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 17:38:58 (1789076338) [10502.729507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10503.593244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10508.772018] LustreError: 346927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10508.800652] LustreError: 346927:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10509.212492] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10513.891806] LustreError: 346928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10519.013908] LustreError: 346927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10519.523039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10520.113119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10525.520128] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10536.444571] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10545.241253] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10554.475306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10554.757122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10562.268506] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10573.064274] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10581.310305] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10583.481547] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10585.741197] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10588.116835] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10590.373163] Lustre: Failing over lustre-MDT0000 [10590.706949] Lustre: server umount lustre-MDT0000 complete [10590.795151] LustreError: 346923:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10591.728118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10591.745678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10593.760640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10593.784571] Lustre: Skipped 1 previous similar message [10595.889558] LustreError: 346924:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10595.907130] LustreError: 346924:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10604.005284] LustreError: 346924:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10604.035419] LustreError: 346924:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [10610.145871] Lustre: 340415:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789076443/real 1789076443] req@0000000062f4801d x1875982301607680/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789076459 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10610.191788] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10612.499682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10620.395049] LustreError: 348025:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10620.435658] LustreError: 348025:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [10621.818473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10621.899648] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10624.078588] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10626.909930] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10627.054325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10627.123983] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10627.177472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10637.548662] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10639.147347] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10642.950845] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10645.070577] Lustre: Failing over lustre-MDT0000 [10647.524504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10647.560900] Lustre: Skipped 1 previous similar message [10647.734953] Lustre: server umount lustre-MDT0000 complete [10652.642186] LustreError: 346928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10652.678482] LustreError: 346928:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [10663.395400] Lustre: 340417:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789076497/real 1789076497] req@000000002fedb8ac x1875982301629440/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789076513 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10663.471667] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10670.675421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10675.263275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10675.352103] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10679.852039] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10680.259067] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10680.327611] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10680.341147] Lustre: Skipped 2 previous similar messages [10680.370208] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10680.440545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10690.574468] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10692.407826] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10700.768835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10700.769220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10700.770111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10700.797342] Lustre: Skipped 2 previous similar messages [10704.827914] Lustre: server umount lustre-MDT0000 complete [10709.924568] LustreError: 346908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789076559 with bad export cookie 15528515119435252507 [10709.943516] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10710.362441] Lustre: server umount lustre-MDT0001 complete [10725.529195] Lustre: server umount lustre-OST0000 complete [10741.510135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10741.991805] LustreError: 353277:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10742.008561] LustreError: 353277:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 27 previous similar messages [10742.089697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10746.568565] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10755.889857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10761.149889] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10769.638576] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10775.790274] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10782.880579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10786.292872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10790.178958] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10799.242680] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10811.872043] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10811.888536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10811.907273] Lustre: Skipped 1 previous similar message [10811.911870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10811.916073] Lustre: Skipped 2 previous similar messages [10823.135832] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10823.352311] Lustre: server umount lustre-OST0000 complete [10828.256189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10828.264290] LustreError: Skipped 1 previous similar message [10834.089243] Lustre: server umount lustre-MDT0000 complete [10838.377786] LustreError: 353259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789076688 with bad export cookie 15528515119435253249 [10838.390554] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10838.391402] LustreError: 353259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10839.006405] Lustre: server umount lustre-MDT0001 complete [10853.276380] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [10856.773552] Key type lgssc unregistered [10857.121656] LNet: 356810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10857.135962] LNetError: 356810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10857.153871] LNet: Removed LNI 192.168.203.137@tcp [10858.153543] Key type .llcrypt unregistered [10858.157866] Key type ._llcrypt unregistered [10888.794960] Key type ._llcrypt registered [10888.797755] Key type .llcrypt registered [10888.893239] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [10889.787806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10889.933067] alg: No test for adler32 (adler32-zlib) [10891.021340] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [10891.301099] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [10893.063211] Key type lgssc registered [10894.272932] Lustre: Echo OBD driver; http://www.lustre.org/ [10902.202715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10902.224047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10903.869257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10909.897784] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10913.260470] LustreError: 358323:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10913.284636] LustreError: 358323:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10918.373663] LustreError: 358325:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10920.039141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10920.544055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10925.412921] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10933.845846] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10940.696357] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10949.225657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10949.894898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10960.805295] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [10970.859944] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10973.459900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10986.984927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10986.999247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10987.022864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10987.489979] Lustre: server umount lustre-MDT0000 complete [10989.037713] LustreError: 358325:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10989.039507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10989.104173] Lustre: Skipped 1 previous similar message [10992.972850] LustreError: 358305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789076842 with bad export cookie 16563600406152400561 [10993.002907] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10993.010769] LustreError: 358305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10993.882786] Lustre: server umount lustre-MDT0001 complete [11009.993528] Lustre: server umount lustre-OST0000 complete [11025.873774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11026.404952] LustreError: 362062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11026.431715] LustreError: 362062:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11026.501184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11031.521646] LustreError: 362063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11033.177822] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11041.765768] LustreError: 362062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11041.786579] LustreError: 362062:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11043.891788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11044.415256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11050.229825] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11060.803857] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11068.879948] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11076.976300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11077.441639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11086.224958] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11095.926305] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11096.884082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11117.536815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11117.541455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11117.557789] Lustre: Skipped 2 previous similar messages [11117.576609] Lustre: Skipped 2 previous similar messages [11122.661405] LustreError: 362063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11122.677457] LustreError: 362063:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11122.823973] Lustre: server umount lustre-MDT0000 complete [11126.806924] LustreError: 362043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789076976 with bad export cookie 16563600406152401674 [11126.815611] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11126.819752] LustreError: 362043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11127.123367] Lustre: server umount lustre-MDT0001 complete [11142.448396] Lustre: server umount lustre-OST0000 complete [11159.713721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11160.132043] LustreError: 365978:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11160.220995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11164.658610] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11173.587899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11178.750923] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11186.760833] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11193.542231] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11201.605460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11202.049380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11202.065780] Lustre: Skipped 1 previous similar message [11206.157803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11209.776464] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11219.165050] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11226.600283] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11226.622499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11226.653268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11228.640549] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11228.652320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11230.771625] Lustre: server umount lustre-OST0000 complete [11239.392495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11239.409026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11239.415090] Lustre: Skipped 2 previous similar messages [11239.438511] Lustre: Skipped 2 previous similar messages [11244.088385] Lustre: server umount lustre-MDT0000 complete [11244.516242] LustreError: 366706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11244.543472] LustreError: 366706:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11249.083232] LustreError: 365958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789077098 with bad export cookie 16563600406152402689 [11249.086482] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11249.102746] LustreError: 365958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11249.831515] Lustre: server umount lustre-MDT0001 complete [11267.676507] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [11272.031267] Key type lgssc unregistered [11272.474521] LNet: 369515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11272.492866] LNetError: 369515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11272.519037] LNet: Removed LNI 192.168.203.137@tcp [11273.990771] Key type .llcrypt unregistered [11273.992495] Key type ._llcrypt unregistered [11307.972174] Key type ._llcrypt registered [11307.975744] Key type .llcrypt registered [11308.116747] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [11309.637766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11309.704735] alg: No test for adler32 (adler32-zlib) [11310.856291] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [11311.239335] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [11312.936334] Key type lgssc registered [11314.201985] Lustre: Echo OBD driver; http://www.lustre.org/ [11322.535603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11322.574707] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11322.585459] LustreError: 370953:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11322.612154] LustreError: 370953:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11322.622142] LustreError: 370953:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11322.627085] LustreError: 370953:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11322.639226] LustreError: 370953:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11330.605056] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11330.617441] LustreError: 371352:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11330.639785] LustreError: 371352:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11330.651623] LustreError: 371352:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11330.659959] LustreError: 371352:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11330.670217] LustreError: 371352:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11351.314844] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [11354.113601] Key type lgssc unregistered [11354.437416] LNet: 372354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11354.449891] LNetError: 372354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11354.480684] LNet: Removed LNI 192.168.203.137@tcp [11355.485588] Key type .llcrypt unregistered [11355.488920] Key type ._llcrypt unregistered [11376.270758] Key type ._llcrypt registered [11376.273688] Key type .llcrypt registered [11376.481741] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [11394.496995] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [11395.860652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11396.049839] alg: No test for adler32 (adler32-zlib) [11397.544243] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [11398.008435] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [11399.943393] Key type lgssc registered [11401.446489] Lustre: Echo OBD driver; http://www.lustre.org/ [11442.595583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11442.623478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11444.187449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11444.263416] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11444.313485] Lustre: lustre-MDT0000: new disk, initializing [11444.374210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11444.387551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11452.547811] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11469.239929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11469.330291] Lustre: 375509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11469.371862] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11469.375225] Lustre: Skipped 1 previous similar message [11469.425925] Lustre: lustre-MDT0001: new disk, initializing [11469.539188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11469.609605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11469.623595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11475.786664] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11488.263972] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11497.131610] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11508.597134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11509.911821] Lustre: lustre-OST0000: new disk, initializing [11509.921513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11509.939733] Lustre: 377480:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11510.142949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11515.508160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11515.531404] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11515.695215] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11521.267758] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11534.402061] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11540.732393] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11541.276790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11545.739536] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11545.989124] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11551.726063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11551.735574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11551.739348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11551.751495] Lustre: Skipped 1 previous similar message [11551.793892] LustreError: Skipped 1 previous similar message [11554.737416] Lustre: server umount lustre-OST0000 complete [11561.952398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11561.958121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11561.972596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11561.976499] Lustre: Skipped 1 previous similar message [11561.985550] Lustre: Skipped 2 previous similar messages [11567.074863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11567.084665] Lustre: Skipped 1 previous similar message [11571.170310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11571.196735] Lustre: Skipped 1 previous similar message [11574.239118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11574.568391] Lustre: server umount lustre-MDT0000 complete [11580.293383] LustreError: 375500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789077430 with bad export cookie 18025825938012279141 [11580.298627] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11580.300528] LustreError: 375500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11580.579410] Lustre: server umount lustre-MDT0001 complete [11594.675194] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11597.227455] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 17:57:24 (1789077444) [11605.184067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11605.498789] LustreError: 379943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11605.519775] LustreError: 379943:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11605.575961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11610.239376] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11610.593424] LustreError: 379944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11615.712480] LustreError: 379943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11619.017970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11619.472555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11624.295927] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11632.682474] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11638.887938] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11646.811909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11647.206547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11654.856669] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11666.039385] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11674.249454] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11674.573463] Lustre: lustre-OST0001: new disk, initializing [11674.580167] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11674.586977] Lustre: 382784:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11674.698189] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11681.692311] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11683.948100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11683.960711] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11684.053336] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11692.826509] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11699.683954] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11699.688807] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11699.702745] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11699.742752] Lustre: Skipped 1 previous similar message [11703.321771] Lustre: server umount lustre-OST0001 complete [11704.804415] LustreError: 381766:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11704.817357] LustreError: 381766:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11709.924153] LustreError: 381768:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11709.942171] LustreError: 381768:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11720.161925] LustreError: 381767:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11720.175399] LustreError: 381767:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11721.185946] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11721.204054] LustreError: Skipped 1 previous similar message [11721.215559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11721.234758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11721.244977] Lustre: Skipped 2 previous similar messages [11727.262702] Lustre: server umount lustre-OST0000 complete [11731.936800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11731.955183] LustreError: Skipped 1 previous similar message [11731.962414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11731.992534] Lustre: Skipped 1 previous similar message [11737.339598] Lustre: server umount lustre-MDT0000 complete [11740.642391] LustreError: 379944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11740.661938] LustreError: 379944:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11741.370775] LustreError: 379923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789077591 with bad export cookie 18025825938012280016 [11741.379770] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11741.380059] LustreError: 379923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11747.800360] Lustre: server umount lustre-MDT0001 complete [11758.211779] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [11761.561667] Key type lgssc unregistered [11761.852637] LNet: 384768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11761.862899] LNetError: 384768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11761.878658] LNet: Removed LNI 192.168.203.137@tcp [11762.826830] Key type .llcrypt unregistered [11762.838809] Key type ._llcrypt unregistered [11819.736576] Key type ._llcrypt registered [11819.739472] Key type .llcrypt registered [11819.830140] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [11821.192743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11821.255319] alg: No test for adler32 (adler32-zlib) [11822.291426] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [11822.653924] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [11824.455299] Key type lgssc registered [11825.782685] Lustre: Echo OBD driver; http://www.lustre.org/ [11836.072862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11836.120089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11837.699793] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11837.745216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11838.047888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11844.741857] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11855.050632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11855.227240] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11855.316570] Lustre: 387578:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11855.809440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11861.259641] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11872.120587] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11881.288229] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11886.561119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11886.578062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11886.596335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11887.094917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11887.142061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11887.171517] Lustre: Skipped 1 previous similar message [11891.246515] Lustre: server umount lustre-MDT0000 complete [11896.069307] LustreError: 387570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789077745 with bad export cookie 15122337879915210574 [11896.071786] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11896.095125] LustreError: 387570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11896.574552] Lustre: server umount lustre-MDT0001 complete [11907.970031] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 18:02:36 (1789077756) [11916.483389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11917.231860] LustreError: 390093:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11917.269257] LustreError: 390093:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11917.336539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11921.790685] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11922.409933] LustreError: 390094:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11927.545452] LustreError: 390093:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11932.647716] LustreError: 390094:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11935.874765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11936.706759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11944.316297] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11956.650575] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11963.982537] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11972.181368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11972.500504] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11972.855887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11980.684915] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [11993.154916] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12002.513635] Lustre: Permanently deactivating lustre-OST0000 [12002.564618] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12010.032107] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12012.680870] LustreError: 392991:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12012.687562] LustreError: 392991:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12012.691712] LustreError: 392991:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12016.252395] LustreError: 393040:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12016.256269] LustreError: 393040:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12021.972289] Lustre: server umount lustre-OST0000 complete [12025.856406] LustreError: 393289:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12025.870121] LustreError: 393289:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12028.901827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12028.917464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12028.949789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12035.042384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12035.044821] Lustre: Skipped 2 previous similar messages [12038.195632] Lustre: server umount lustre-MDT0000 complete [12040.160647] LustreError: 390093:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12040.177548] LustreError: 390093:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12043.627796] LustreError: 390074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789077893 with bad export cookie 15122337879915211155 [12043.641522] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12043.659023] LustreError: 390074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12054.478254] Lustre: server umount lustre-MDT0001 complete [12067.315607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12072.432699] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12075.670577] LustreError: 394449:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12081.625637] Lustre: 394788:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.137@tcp [12083.938630] Lustre: server umount MGS complete [12095.466600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12096.057723] LustreError: 395511:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12096.080863] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12096.094025] Lustre: Skipped 1 previous similar message [12096.171306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12100.482513] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12108.507655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12108.834079] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12113.724409] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12123.414719] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12129.709105] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12132.021584] Lustre: Permanently reactivating lustre-OST0000 [12132.032906] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12132.037563] Lustre: Skipped 2 previous similar messages [12158.950861] LustreError: 397274:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12167.244085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12167.593026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12167.605930] Lustre: Skipped 1 previous similar message [12174.368177] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12181.906948] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12189.158508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12189.171040] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12189.185699] Lustre: Skipped 1 previous similar message [12189.197463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12189.201460] LustreError: Skipped 1 previous similar message [12189.230191] Lustre: Skipped 1 previous similar message [12189.778947] Lustre: server umount lustre-OST0000 complete [12194.273324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12194.285779] Lustre: Skipped 1 previous similar message [12199.402767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12199.420256] Lustre: Skipped 3 previous similar messages [12200.573627] Lustre: server umount lustre-MDT0000 complete [12204.209409] LustreError: 395494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789078054 with bad export cookie 15122337879915212205 [12204.219061] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12204.226920] LustreError: 395494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12204.528596] Lustre: server umount lustre-MDT0001 complete [12216.407849] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [12219.554713] Key type lgssc unregistered [12219.846749] LNet: 399877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12219.867269] LNetError: 399877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12220.908230] LNet: Removed LNI 192.168.203.137@tcp [12221.909375] Key type .llcrypt unregistered [12221.912274] Key type ._llcrypt unregistered [12249.937590] Key type ._llcrypt registered [12249.939751] Key type .llcrypt registered [12250.077191] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [12273.011548] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [12274.888525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12275.019465] alg: No test for adler32 (adler32-zlib) [12276.109518] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [12276.322703] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [12278.063277] Key type lgssc registered [12279.458752] Lustre: Echo OBD driver; http://www.lustre.org/ [12321.964766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12322.015868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12323.577396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12323.648300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12323.862889] Lustre: lustre-MDT0000: new disk, initializing [12324.015057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12324.059693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12330.093737] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12341.773635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12341.946743] Lustre: 403053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12342.056702] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12342.067178] Lustre: Skipped 1 previous similar message [12342.199414] Lustre: lustre-MDT0001: new disk, initializing [12342.378793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12342.423957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12342.441799] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12348.176553] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12359.276542] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12366.269510] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12373.585380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12374.013208] Lustre: lustre-OST0000: new disk, initializing [12374.017701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12374.025956] Lustre: 405025:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12374.145506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12381.861876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12381.879922] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12382.148625] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12382.309876] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12393.696865] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12399.210507] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12399.569378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12405.049607] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12405.310199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12407.776751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12407.788903] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12407.810223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12407.833660] Lustre: Skipped 1 previous similar message [12412.904988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12412.912623] Lustre: Skipped 1 previous similar message [12418.016147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12422.112749] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12422.298118] Lustre: server umount lustre-OST0000 complete [12428.265044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12428.280769] Lustre: Skipped 1 previous similar message [12428.286690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12428.295455] Lustre: Skipped 2 previous similar messages [12432.742401] Lustre: server umount lustre-MDT0000 complete [12437.139026] LustreError: 403862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789078286 with bad export cookie 6036700946160404332 [12437.147736] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12437.150485] LustreError: 403862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12437.636143] Lustre: server umount lustre-MDT0001 complete [12446.504046] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12448.387475] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 18:11:36 (1789078296) [12456.842547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12457.398492] LustreError: 407486:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12457.420359] LustreError: 407486:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12457.515964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12462.564585] LustreError: 407486:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12463.160746] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12467.681116] LustreError: 407487:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12472.802430] LustreError: 407486:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12473.036345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12473.627710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12478.512253] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12487.198055] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12493.166243] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12500.748183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12501.173816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12508.103903] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12516.656116] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12518.486598] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12521.123986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12528.098600] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12528.113728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12528.130459] Lustre: Skipped 1 previous similar message [12528.140108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12529.153165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12533.866218] Lustre: server umount lustre-OST0000 complete [12539.365555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12539.397910] Lustre: Skipped 1 previous similar message [12539.411308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12539.414233] Lustre: Skipped 2 previous similar messages [12545.876681] Lustre: server umount lustre-MDT0000 complete [12549.611581] LustreError: 410171:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12549.640621] LustreError: 410171:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12550.254088] LustreError: 407466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789078400 with bad export cookie 6036700946160405214 [12550.269304] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12550.278410] LustreError: 407466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12550.697860] Lustre: server umount lustre-MDT0001 complete [12565.569078] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [12569.063671] Key type lgssc unregistered [12569.383189] LNet: 411209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12569.394186] LNetError: 411209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12569.420254] LNet: Removed LNI 192.168.203.137@tcp [12570.452532] Key type .llcrypt unregistered [12570.454218] Key type ._llcrypt unregistered [12599.934655] Key type ._llcrypt registered [12599.940677] Key type .llcrypt registered [12600.143125] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [12600.728404] Key type .llcrypt unregistered [12600.730048] Key type ._llcrypt unregistered [12621.249482] Key type ._llcrypt registered [12621.251797] Key type .llcrypt registered [12621.416475] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [12623.228371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12623.269648] alg: No test for adler32 (adler32-zlib) [12624.424393] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [12624.665103] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [12626.312655] Key type lgssc registered [12627.743623] Lustre: Echo OBD driver; http://www.lustre.org/ [12636.922115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12636.949488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12638.520662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12643.226525] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12648.936366] LustreError: 413420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12648.952061] LustreError: 413420:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12651.788907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12652.169095] LustreError: 413421:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12652.189845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12657.643588] LustreError: 413420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12657.664973] LustreError: 413420:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12658.831912] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12662.758994] LustreError: 413421:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12662.778959] LustreError: 413421:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12663.799253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12666.293934] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12668.906822] LustreError: 413420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12668.923639] LustreError: 413420:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12673.709090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12674.023853] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12674.031329] Lustre: Skipped 1 previous similar message [12674.178968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12679.286611] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12694.496227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12694.497393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12694.508750] LustreError: Skipped 1 previous similar message [12694.509707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12694.530642] Lustre: Skipped 1 previous similar message [12696.078532] Lustre: server umount lustre-OST0000 complete [12700.131978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12700.150996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12700.158273] Lustre: Skipped 1 previous similar message [12700.183140] Lustre: Skipped 4 previous similar messages [12705.248846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12705.258497] Lustre: Skipped 1 previous similar message [12706.489779] Lustre: server umount lustre-MDT0000 complete [12710.370377] LustreError: 413420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12710.387922] LustreError: 413420:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12710.920223] LustreError: 413399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789078560 with bad export cookie 689447699265093440 [12710.935316] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12710.940939] LustreError: 413399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12711.504389] Lustre: server umount lustre-MDT0001 complete [12725.091118] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [12728.430490] Key type lgssc unregistered [12728.783715] LNet: 416566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12728.789809] LNetError: 416566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12728.816983] LNet: Removed LNI 192.168.203.137@tcp [12730.105633] Key type .llcrypt unregistered [12730.113481] Key type ._llcrypt unregistered [12758.886799] Key type ._llcrypt registered [12758.889800] Key type .llcrypt registered [12759.012660] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [12759.973427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12760.001411] alg: No test for adler32 (adler32-zlib) [12761.275644] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [12761.660236] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [12763.359375] Key type lgssc registered [12764.801472] Lustre: Echo OBD driver; http://www.lustre.org/ [12773.633105] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12773.648420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12789.218187] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12794.335479] LustreError: 417955:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.137@tcp: failed processing log, type 4: rc = -110 [12821.087634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12828.688510] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12837.579983] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12845.254761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12845.821847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12845.910274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12851.192272] LustreError: 417981:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12851.251488] LustreError: 417981:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12851.891822] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12855.276941] LustreError: 417980:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12855.306325] LustreError: 417980:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12860.387950] LustreError: 418992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12860.446688] LustreError: 418992:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12862.933635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12863.496588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12869.502837] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12879.036333] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12885.142484] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12894.177310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12894.182493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12894.196463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12894.220142] Lustre: Skipped 1 previous similar message [12894.241636] Lustre: Skipped 1 previous similar message [12895.045450] Lustre: server umount lustre-OST0000 complete [12899.296791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12899.299239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12899.308168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12899.310266] Lustre: Skipped 1 previous similar message [12901.277040] Lustre: server umount lustre-MDT0000 complete [12904.417645] LustreError: 418997:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12905.336314] LustreError: 417962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789078755 with bad export cookie 4149359743671872369 [12905.351853] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12905.355220] LustreError: 417962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12905.850065] Lustre: server umount lustre-MDT0001 complete [12916.748903] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [12920.400884] Key type lgssc unregistered [12920.862251] LNet: 421526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12920.874483] LNetError: 421526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12921.962313] LNet: Removed LNI 192.168.203.137@tcp [12923.137132] Key type .llcrypt unregistered [12923.142853] Key type ._llcrypt unregistered [12952.591062] Key type ._llcrypt registered [12952.598493] Key type .llcrypt registered [12952.822211] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [12953.958343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12954.217052] alg: No test for adler32 (adler32-zlib) [12955.516552] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [12955.984682] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [12957.711764] Key type lgssc registered [12959.375085] Lustre: Echo OBD driver; http://www.lustre.org/ [12969.742809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12969.761075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12971.398923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12976.723825] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [12981.739026] LustreError: 422940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12981.755749] LustreError: 422940:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12986.817692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12986.862775] LustreError: 422941:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12987.324250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12992.999544] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13001.821738] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13008.302376] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13016.377612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13016.921210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13017.984081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13024.008923] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13033.071532] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13037.545171] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13037.549282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13037.574996] Lustre: Skipped 1 previous similar message [13037.592490] Lustre: Skipped 2 previous similar messages [13039.701352] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.37@tcp (stopping) [13042.672672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13042.685567] Lustre: Skipped 2 previous similar messages [13043.199488] Lustre: server umount lustre-MDT0000 complete [13044.791283] LustreError: 422937:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13046.101420] LustreError: 422920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.37@tcp arrived at 1789078895 with bad export cookie 17705064741550198944 [13046.117418] LustreError: 422920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13047.778805] LustreError: 425074:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13047.807967] LustreError: 425074:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13048.204907] LustreError: 422921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789078898 with bad export cookie 17705064741550198041 [13048.218310] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13052.897055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13052.897384] LustreError: 422937:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13052.913371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13052.913379] Lustre: Skipped 1 previous similar message [13052.914700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13053.006683] LustreError: 422937:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13054.540856] Lustre: server umount lustre-OST0000 complete [13061.798563] Lustre: server umount lustre-MDT0001 complete [13073.010760] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [13075.733537] Key type lgssc unregistered [13076.067780] LNet: 426625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13076.075948] LNetError: 426625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13076.096493] LNet: Removed LNI 192.168.203.137@tcp [13076.926188] Key type .llcrypt unregistered [13076.929121] Key type ._llcrypt unregistered [13103.331416] Key type ._llcrypt registered [13103.343162] Key type .llcrypt registered [13103.494824] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [13104.679016] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13104.954860] alg: No test for adler32 (adler32-zlib) [13106.103944] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [13106.356114] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [13108.071242] Key type lgssc registered [13109.278497] Lustre: Echo OBD driver; http://www.lustre.org/ [13118.292102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13118.327423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13119.986105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13123.994076] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13129.186709] LustreError: 428040:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13129.210326] LustreError: 428040:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13133.512932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13134.146170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13138.898530] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13148.506227] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13155.599419] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13163.322739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13163.596675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13164.706780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13170.997637] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13178.785768] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13185.517297] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13185.525132] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13185.546493] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13187.553993] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13187.556240] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13187.568449] Lustre: Skipped 1 previous similar message [13188.899629] Lustre: server umount lustre-MDT0001 complete [13190.702364] LustreError: 429214:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13192.681788] LustreError: 428035:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13192.711943] LustreError: 428035:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13195.753739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13195.757066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13195.774756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13195.779312] Lustre: Skipped 1 previous similar message [13197.792888] LustreError: 428041:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13197.796275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13202.034503] Lustre: server umount lustre-OST0000 complete [13202.912568] LustreError: 428040:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13205.969587] Lustre: server umount lustre-MDT0000 complete [13219.850645] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [13222.772490] Key type lgssc unregistered [13223.147628] LNet: 431726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13223.158746] LNetError: 431726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13223.172859] LNet: Removed LNI 192.168.203.137@tcp [13224.180925] Key type .llcrypt unregistered [13224.183557] Key type ._llcrypt unregistered [13250.139613] Key type ._llcrypt registered [13250.141811] Key type .llcrypt registered [13250.248939] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [13265.691683] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [13267.001261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13267.161020] alg: No test for adler32 (adler32-zlib) [13268.284816] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [13268.572719] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [13270.279826] Key type lgssc registered [13271.536732] Lustre: Echo OBD driver; http://www.lustre.org/ [13309.653069] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [13326.373762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13326.389579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13327.710919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13327.750848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13327.860646] Lustre: lustre-MDT0000: new disk, initializing [13327.977069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13327.994974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13333.120209] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13351.796448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13352.091432] Lustre: lustre-OST0000: new disk, initializing [13352.097247] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13352.101727] Lustre: Skipped 1 previous similar message [13352.106564] Lustre: 436762:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13352.181421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13358.222814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13358.235956] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13358.271144] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13358.599866] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13367.318221] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13383.797378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13383.905695] Lustre: 435745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13383.984904] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13384.098216] Lustre: lustre-MDT0001: new disk, initializing [13384.222158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13384.292274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13384.305186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13388.863977] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13398.638556] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13398.875939] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13403.680230] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13403.928118] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13409.764733] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13409.774396] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13409.779237] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13409.790266] Lustre: Skipped 1 previous similar message [13409.809680] Lustre: Skipped 1 previous similar message [13410.919413] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.37@tcp (stopping) [13414.893527] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13416.100708] Lustre: server umount lustre-MDT0001 complete [13425.128864] LustreError: 435750:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13425.145808] LustreError: 435750:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [13430.245113] LustreError: 435756:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13430.269249] LustreError: 435756:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13435.360876] LustreError: 435755:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13435.363262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13435.398794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13435.403437] Lustre: Skipped 1 previous similar message [13440.480703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13440.485447] LustreError: 435756:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13445.604536] LustreError: 435755:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13448.159468] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13448.344807] Lustre: server umount lustre-OST0000 complete [13459.078839] Lustre: server umount lustre-MDT0000 complete [13471.383535] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [13473.995975] Key type lgssc unregistered [13474.358258] LNet: 440514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13474.370186] LNetError: 440514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13474.393834] LNet: Removed LNI 192.168.203.137@tcp [13475.126234] Key type .llcrypt unregistered [13475.129855] Key type ._llcrypt unregistered [13513.558758] Key type ._llcrypt registered [13513.561768] Key type .llcrypt registered [13513.659230] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [13513.965378] Key type .llcrypt unregistered [13513.966962] Key type ._llcrypt unregistered [13530.383462] Key type ._llcrypt registered [13530.386249] Key type .llcrypt registered [13530.495504] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [13531.495642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13531.662471] alg: No test for adler32 (adler32-zlib) [13532.823667] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [13533.161037] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [13534.919329] Key type lgssc registered [13536.069736] Lustre: Echo OBD driver; http://www.lustre.org/ [13563.372028] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13563.394414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13564.588712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13564.625363] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13564.696608] Lustre: lustre-MDT0000: new disk, initializing [13564.779749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13564.821468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13569.140396] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13580.491356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13580.721252] Lustre: 443883:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13580.831385] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13580.838790] Lustre: Skipped 1 previous similar message [13580.969828] Lustre: lustre-MDT0001: new disk, initializing [13581.105047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13581.173768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13581.186425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13586.474643] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13596.887086] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13604.611255] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13613.277660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13613.649817] Lustre: lustre-OST0000: new disk, initializing [13613.653599] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13613.660431] Lustre: 445860:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13613.749582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13621.421440] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13621.859111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13621.874891] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13622.083169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13633.296968] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13642.723186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13642.729040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13642.748311] Lustre: Skipped 1 previous similar message [13642.766588] Lustre: Skipped 2 previous similar messages [13647.678121] Lustre: server umount lustre-MDT0000 complete [13651.581765] LustreError: 444681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789079501 with bad export cookie 17580915987949335581 [13651.591633] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13651.616981] LustreError: 444681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13651.895437] Lustre: server umount lustre-MDT0001 complete [13665.442563] Lustre: server umount lustre-OST0000 complete [13677.216641] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 18:32:05 (1789079525) [13684.615298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13685.184266] LustreError: 447886:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13685.217413] LustreError: 447886:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13685.285255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13689.907473] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13690.342275] LustreError: 447887:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13695.459381] LustreError: 447886:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13698.313274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13698.767778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13703.756019] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13712.178136] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13718.798839] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13739.255286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13739.602116] Lustre: Found index 0 for lustre-OST0000, updating log [13739.854455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13747.537852] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [13756.226670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13757.352548] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13764.577141] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13764.593103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13764.622951] Lustre: Skipped 1 previous similar message [13764.654901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13765.733720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13765.737772] Lustre: Skipped 1 previous similar message [13768.966974] Lustre: server umount lustre-OST0000 complete [13786.082139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13786.087995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13786.099421] Lustre: Skipped 1 previous similar message [13786.111130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13786.113574] Lustre: Skipped 1 previous similar message [13792.212456] Lustre: server umount lustre-MDT0000 complete [13792.736517] LustreError: 447883:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13792.760818] LustreError: 447883:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13796.490775] LustreError: 447868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789079646 with bad export cookie 17580915987949336974 [13796.520417] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13797.017415] Lustre: server umount lustre-MDT0001 complete [13806.166873] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 18:34:14 (1789079654) [13810.655255] LNet: 1 local NIs in recovery (showing 1): 192.168.203.137@tcp [14430.723134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14431.314047] LustreError: 454298:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14431.344030] LustreError: 454298:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [14431.456944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14436.599750] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [14436.832914] LustreError: 454299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14441.958924] LustreError: 454298:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14446.817923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14447.252768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14452.337918] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [14460.845919] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14467.000563] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14487.910288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14488.115721] Lustre: Found index 0 for lustre-OST0000, updating log [14488.438242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14488.545286] LNet: 1 local NIs in recovery (showing 1): 192.168.203.137@tcp [14495.374551] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [14503.355950] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14505.952738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14505.958470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14505.977379] Lustre: Skipped 1 previous similar message [14505.984500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14506.001591] Lustre: Skipped 2 previous similar messages [14507.848165] Lustre: server umount lustre-OST0000 complete [14512.418521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14512.424130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14512.432905] Lustre: Skipped 1 previous similar message [14512.442558] Lustre: Skipped 1 previous similar message [14513.441744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14513.444792] Lustre: Skipped 1 previous similar message [14515.490371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14515.501279] Lustre: Skipped 3 previous similar messages [14517.867897] Lustre: server umount lustre-MDT0000 complete [14518.561732] LustreError: 454299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14518.590836] LustreError: 454299:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [14521.390186] LustreError: 454280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789080371 with bad export cookie 17580915987949338059 [14521.392129] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14521.409843] LustreError: 454280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [14521.723861] Lustre: server umount lustre-MDT0001 complete [14548.964158] LNet: 1 local NIs in recovery (showing 1): 192.168.203.137@tcp [14552.149719] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [14564.609155] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [14608.582812] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 18:47:36 (1789080456) [14629.264541] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [14633.280643] Key type lgssc unregistered [14633.639760] LNet: 460908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14633.648368] LNetError: 460908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14751.469323] LNet: Removed LNI 192.168.203.137@tcp [14752.560736] Key type .llcrypt unregistered [14752.562895] Key type ._llcrypt unregistered [14772.631911] Key type ._llcrypt registered [14772.635958] Key type .llcrypt registered [14772.787049] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [14774.084750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14774.164105] alg: No test for adler32 (adler32-zlib) [14775.256732] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [14775.650560] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [14777.318496] Key type lgssc registered [14778.447288] Lustre: Echo OBD driver; http://www.lustre.org/ [14782.897192] LNet: Added LNI 192.168.203.137@tcp100 [8/256/0/180] [14783.722726] LNet: Added LNI 192.168.203.137@tcp101 [8/256/0/180] [14784.717414] LNet: Added LNI 192.168.203.137@tcp102 [8/256/0/180] [14785.747538] LNet: Added LNI 192.168.203.137@tcp103 [8/256/0/180] [14786.442840] LNet: Added LNI 192.168.203.137@tcp104 [8/256/0/180] [14787.215747] LNet: Added LNI 192.168.203.137@tcp105 [8/256/0/180] [14787.936497] LNet: Added LNI 192.168.203.137@tcp106 [8/256/0/180] [14788.508182] LNet: Added LNI 192.168.203.137@tcp107 [8/256/0/180] [14789.200941] LNet: Added LNI 192.168.203.137@tcp108 [8/256/0/180] [14790.053809] LNet: Added LNI 192.168.203.137@tcp109 [8/256/0/180] [14790.919123] LNet: Added LNI 192.168.203.137@tcp110 [8/256/0/180] [14791.728267] LNet: Added LNI 192.168.203.137@tcp111 [8/256/0/180] [14792.639385] LNet: Added LNI 192.168.203.137@tcp112 [8/256/0/180] [14793.596111] LNet: Added LNI 192.168.203.137@tcp113 [8/256/0/180] [14794.595780] LNet: Added LNI 192.168.203.137@tcp114 [8/256/0/180] [14795.633456] LNet: Added LNI 192.168.203.137@tcp115 [8/256/0/180] [14796.371852] LNet: Added LNI 192.168.203.137@tcp116 [8/256/0/180] [14797.360715] LNet: Added LNI 192.168.203.137@tcp117 [8/256/0/180] [14798.400921] LNet: Added LNI 192.168.203.137@tcp118 [8/256/0/180] [14799.361787] LNet: Added LNI 192.168.203.137@tcp119 [8/256/0/180] [14800.351686] LNet: Added LNI 192.168.203.137@tcp120 [8/256/0/180] [14801.318650] LNet: Added LNI 192.168.203.137@tcp121 [8/256/0/180] [14802.087617] LNet: Added LNI 192.168.203.137@tcp122 [8/256/0/180] [14802.757927] LNet: Added LNI 192.168.203.137@tcp123 [8/256/0/180] [14803.566698] LNet: Added LNI 192.168.203.137@tcp124 [8/256/0/180] [14804.485123] LNet: Added LNI 192.168.203.137@tcp125 [8/256/0/180] [14805.478929] LNet: Added LNI 192.168.203.137@tcp126 [8/256/0/180] [14806.317787] LNet: Added LNI 192.168.203.137@tcp127 [8/256/0/180] [14807.073229] LNet: Added LNI 192.168.203.137@tcp128 [8/256/0/180] [14807.943361] LNet: Added LNI 192.168.203.137@tcp129 [8/256/0/180] [14808.768291] LNet: Added LNI 192.168.203.137@tcp130 [8/256/0/180] [14809.583589] LNet: Added LNI 192.168.203.137@tcp131 [8/256/0/180] [14810.510461] LNet: Added LNI 192.168.203.137@tcp132 [8/256/0/180] [14811.360347] LNet: Added LNI 192.168.203.137@tcp133 [8/256/0/180] [14812.146396] LNet: Added LNI 192.168.203.137@tcp134 [8/256/0/180] [14812.858558] LNet: Added LNI 192.168.203.137@tcp135 [8/256/0/180] [14819.466083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14819.511609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14820.617354] Lustre: 463938:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.137@tcp: can't add NID 192.168.203.137@tcp130: rc = -75 [14820.729357] LNetError: 461398:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.137 [14820.759982] LNetError: Protocol error connecting to 192.168.203.137@tcp122 at host 192.168.203.137:988: is it running a compatible version of Lustre? [14820.788282] LNetError: Skipped 8 previous similar messages [14820.855409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14820.891660] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14821.027569] Lustre: lustre-MDT0000: new disk, initializing [14821.152570] Lustre: 463991:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.137@tcp: can't add NID 192.168.203.137@tcp130: rc = -75 [14821.164013] Lustre: 463991:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14821.183872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14821.200592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14821.731210] LNet: 6 local NIs in recovery (showing 6): 192.168.203.137@tcp125, 192.168.203.137@tcp126, 192.168.203.137@tcp128, 192.168.203.137@tcp129, 192.168.203.137@tcp127, 192.168.203.137@tcp122 [14821.733466] LNetError: 463964:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.137 [14821.765776] LNetError: 463964:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [14821.781300] LNetError: Protocol error connecting to 192.168.203.137@tcp104 at host 192.168.203.137:988: is it running a compatible version of Lustre? [14821.798956] LNetError: Skipped 1 previous similar message [14825.528043] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [14835.156897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14835.288791] Lustre: 463955:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14835.343881] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14835.348029] Lustre: Skipped 1 previous similar message [14835.401503] Lustre: lustre-MDT0001: new disk, initializing [14835.436383] Lustre: 464753:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.137@tcp: can't add NID 192.168.203.137@tcp130: rc = -75 [14835.443708] Lustre: 464753:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14835.520888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14835.548727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14835.559074] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14838.002187] Lustre: 464916:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.137@tcp: can't add NID 192.168.203.137@tcp130: rc = -75 [14838.011158] Lustre: 464916:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14839.821278] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [14849.810846] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14850.400853] LNetError: 461396:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.137 [14850.411209] LNetError: 461396:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14850.425197] LNetError: Protocol error connecting to 192.168.203.137@tcp104 at host 192.168.203.137:988: is it running a compatible version of Lustre? [14854.497052] LNetError: 463962:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.137 [14854.523625] LNetError: Protocol error connecting to 192.168.203.137@tcp123 at host 192.168.203.137:988: is it running a compatible version of Lustre? [14855.209719] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14858.603765] LNetError: 464022:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.137 [14858.618814] LNetError: 464022:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [14858.625816] LNetError: Protocol error connecting to 192.168.203.137@tcp124 at host 192.168.203.137:988: is it running a compatible version of Lustre? [14858.635929] LNetError: Skipped 2 previous similar messages [14861.003930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14861.268983] Lustre: lustre-OST0000: new disk, initializing [14861.276247] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14861.282754] Lustre: 465926:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14861.348702] Lustre: 465931:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.137@tcp: can't add NID 192.168.203.137@tcp130: rc = -75 [14861.355678] Lustre: 465931:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14861.407272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14866.542568] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [14866.788950] LNetError: 463958:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.137 [14866.804518] LNetError: 463958:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14866.809630] LNetError: Protocol error connecting to 192.168.203.137@tcp123 at host 192.168.203.137:988: is it running a compatible version of Lustre? [14866.819405] LNetError: Skipped 1 previous similar message [14868.553796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14868.568731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14868.699180] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14875.883085] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14884.199760] LNetError: 463961:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.137 [14884.218516] LNetError: 463961:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14884.232847] LNetError: Protocol error connecting to 192.168.203.137@tcp124 at host 192.168.203.137:988: is it running a compatible version of Lustre? [14884.247429] LNetError: Skipped 7 previous similar messages [14884.322585] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14884.324284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14884.333144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14884.345073] Lustre: Skipped 1 previous similar message [14884.358052] Lustre: Skipped 1 previous similar message [14888.800909] Lustre: server umount lustre-OST0000 complete [14894.561052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14894.570510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14894.577450] Lustre: Skipped 1 previous similar message [14894.588506] Lustre: Skipped 1 previous similar message [14899.005610] Lustre: server umount lustre-MDT0000 complete [14902.729945] LustreError: 463946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789080752 with bad export cookie 1630265158499255047 [14902.734466] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14902.743263] LustreError: 463946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14903.053973] Lustre: server umount lustre-MDT0001 complete [14914.810943] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [14917.787146] Key type lgssc unregistered [14918.176647] LNet: 467837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14918.184518] LNetError: 467837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14918.191928] LNet: Removed LNI 192.168.203.137@tcp [14919.199176] LNet: Removed LNI 192.168.203.137@tcp100 [14919.204883] LNetError: 467837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14919.220548] LNetError: 467837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14919.226468] LNet: Removed LNI 192.168.203.137@tcp101 [14920.287173] LNet: Removed LNI 192.168.203.137@tcp102 [14920.301208] LNetError: 467837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14920.312655] LNetError: 467837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14920.319885] LNet: Removed LNI 192.168.203.137@tcp103 [14921.376894] LNet: Removed LNI 192.168.203.137@tcp104 [14922.399179] LNet: Removed LNI 192.168.203.137@tcp105 [14922.408484] LNetError: 467837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14922.430543] LNetError: 467837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14922.446688] LNet: Removed LNI 192.168.203.137@tcp106 [14923.487493] LNet: Removed LNI 192.168.203.137@tcp107 [14924.511169] LNet: Removed LNI 192.168.203.137@tcp108 [14925.535142] LNet: Removed LNI 192.168.203.137@tcp109 [14926.559407] LNet: Removed LNI 192.168.203.137@tcp110 [14926.567579] LNetError: 467837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14926.578526] LNetError: 467837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14926.584727] LNet: Removed LNI 192.168.203.137@tcp111 [14927.650338] LNet: Removed LNI 192.168.203.137@tcp112 [14928.671700] LNet: Removed LNI 192.168.203.137@tcp113 [14929.695522] LNet: Removed LNI 192.168.203.137@tcp114 [14930.719455] LNet: Removed LNI 192.168.203.137@tcp115 [14931.750099] LNet: Removed LNI 192.168.203.137@tcp116 [14932.767304] LNet: Removed LNI 192.168.203.137@tcp117 [14933.796352] LNet: Removed LNI 192.168.203.137@tcp118 [14934.815387] LNet: Removed LNI 192.168.203.137@tcp119 [14934.819231] LNetError: 467837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14934.840251] LNetError: 467837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14934.847340] LNet: Removed LNI 192.168.203.137@tcp120 [14935.903152] LNet: Removed LNI 192.168.203.137@tcp121 [14936.928599] LNet: Removed LNI 192.168.203.137@tcp122 [14937.952740] LNet: Removed LNI 192.168.203.137@tcp123 [14938.975274] LNet: Removed LNI 192.168.203.137@tcp124 [14939.999154] LNet: Removed LNI 192.168.203.137@tcp125 [14941.023158] LNet: Removed LNI 192.168.203.137@tcp126 [14942.047133] LNet: Removed LNI 192.168.203.137@tcp127 [14943.073115] LNet: Removed LNI 192.168.203.137@tcp128 [14944.095940] LNet: Removed LNI 192.168.203.137@tcp129 [14944.108127] LNet: Removed LNI 192.168.203.137@tcp130 [14944.114783] LNet: Removed LNI 192.168.203.137@tcp131 [14944.121196] LNet: Removed LNI 192.168.203.137@tcp132 [14944.129629] LNet: Removed LNI 192.168.203.137@tcp133 [14944.140339] LNet: Removed LNI 192.168.203.137@tcp134 [14944.151167] LNet: Removed LNI 192.168.203.137@tcp135 [14945.273606] Key type .llcrypt unregistered [14945.277174] Key type ._llcrypt unregistered [14971.204195] Key type ._llcrypt registered [14971.206272] Key type .llcrypt registered [14971.333698] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [14988.249209] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [14989.385704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14989.463517] alg: No test for adler32 (adler32-zlib) [14990.527744] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [14990.764337] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [14992.447795] Key type lgssc registered [14993.470232] Lustre: Echo OBD driver; http://www.lustre.org/ [15025.282177] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 18:54:33 (1789080873) [15045.834111] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [15049.136971] Key type lgssc unregistered [15049.438732] LNet: 471920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15049.445774] LNetError: 471920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15049.469377] LNet: Removed LNI 192.168.203.137@tcp [15050.434572] Key type .llcrypt unregistered [15050.439093] Key type ._llcrypt unregistered [15068.426733] Key type ._llcrypt registered [15068.430160] Key type .llcrypt registered [15068.570573] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [15069.691629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15069.779067] alg: No test for adler32 (adler32-zlib) [15070.895923] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [15071.260312] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [15072.986378] Key type lgssc registered [15073.996547] Lustre: Echo OBD driver; http://www.lustre.org/ [15084.254911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15084.285878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15085.624475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15085.673641] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15085.774525] Lustre: lustre-MDT0000: new disk, initializing [15085.889294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15085.916284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15090.482623] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [15101.316977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15101.484424] Lustre: 473129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15101.572083] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15101.579619] Lustre: Skipped 1 previous similar message [15101.674601] Lustre: lustre-MDT0001: new disk, initializing [15101.791372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15101.840160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15101.851772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15106.338742] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [15116.190493] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15122.496352] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15130.581234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15130.988254] Lustre: lustre-OST0000: new disk, initializing [15130.991984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15131.013959] Lustre: 475101:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15131.142887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15139.080497] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [15141.098520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15141.110694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15141.210210] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15150.282806] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15154.368675] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing lnet_if_list [15161.808881] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15161.892531] LNet: Added LNI 192.168.203.137@tcp10 [8/256/0/180] [15168.447184] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15168.509753] LNet: Added LNI 192.168.203.137@tcp11 [8/256/0/180] [15174.054524] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15174.111476] LNet: Added LNI 192.168.203.137@tcp12 [8/256/0/180] [15180.247451] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15180.291260] LNet: Added LNI 192.168.203.137@tcp13 [8/256/0/180] [15186.137468] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15186.194920] LNet: Added LNI 192.168.203.137@tcp14 [8/256/0/180] [15193.155742] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15193.178534] LNet: Added LNI 192.168.203.137@tcp15 [8/256/0/180] [15199.654498] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15199.688537] LNet: Added LNI 192.168.203.137@tcp16 [8/256/0/180] [15206.137869] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15206.211415] LNet: Added LNI 192.168.203.137@tcp17 [8/256/0/180] [15212.833391] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15212.871831] LNet: Added LNI 192.168.203.137@tcp18 [8/256/0/180] [15218.491247] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15218.523391] LNet: Added LNI 192.168.203.137@tcp19 [8/256/0/180] [15225.137228] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15225.181561] LNet: Added LNI 192.168.203.137@tcp20 [8/256/0/180] [15231.086869] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15231.143203] LNet: Added LNI 192.168.203.137@tcp21 [8/256/0/180] [15237.238589] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15237.318730] LNet: Added LNI 192.168.203.137@tcp22 [8/256/0/180] [15243.526423] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15243.565269] LNet: Added LNI 192.168.203.137@tcp23 [8/256/0/180] [15249.997371] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15250.025362] LNet: Added LNI 192.168.203.137@tcp24 [8/256/0/180] [15256.088415] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15256.110478] LNet: Added LNI 192.168.203.137@tcp25 [8/256/0/180] [15262.162174] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15262.188865] LNet: Added LNI 192.168.203.137@tcp26 [8/256/0/180] [15268.425543] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15268.461915] LNet: Added LNI 192.168.203.137@tcp27 [8/256/0/180] [15274.826985] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15274.877604] LNet: Added LNI 192.168.203.137@tcp28 [8/256/0/180] [15282.577455] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15282.625677] LNet: Added LNI 192.168.203.137@tcp29 [8/256/0/180] [15289.079497] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15289.126358] LNet: Added LNI 192.168.203.137@tcp30 [8/256/0/180] [15297.473679] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15297.510262] LNet: Added LNI 192.168.203.137@tcp31 [8/256/0/180] [15304.296517] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15304.357993] LNet: Added LNI 192.168.203.137@tcp32 [8/256/0/180] [15313.158432] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15313.199977] LNet: Added LNI 192.168.203.137@tcp33 [8/256/0/180] [15321.658953] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15321.691438] LNet: Added LNI 192.168.203.137@tcp34 [8/256/0/180] [15329.935190] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15329.966948] LNet: Added LNI 192.168.203.137@tcp35 [8/256/0/180] [15338.389165] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15338.415648] LNet: Added LNI 192.168.203.137@tcp36 [8/256/0/180] [15346.313916] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15346.348110] LNet: Added LNI 192.168.203.137@tcp37 [8/256/0/180] [15353.924101] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15353.983179] LNet: Added LNI 192.168.203.137@tcp38 [8/256/0/180] [15362.976169] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15363.025065] LNet: Added LNI 192.168.203.137@tcp39 [8/256/0/180] [15373.111703] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15373.164711] LNet: Added LNI 192.168.203.137@tcp40 [8/256/0/180] [15382.949883] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15383.009389] LNet: Added LNI 192.168.203.137@tcp41 [8/256/0/180] [15393.613873] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15393.675789] LNet: Added LNI 192.168.203.137@tcp42 [8/256/0/180] [15403.479563] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15403.521775] LNet: Added LNI 192.168.203.137@tcp43 [8/256/0/180] [15413.072924] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15413.095579] LNet: Added LNI 192.168.203.137@tcp44 [8/256/0/180] [15421.408834] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15421.438388] LNet: Added LNI 192.168.203.137@tcp45 [8/256/0/180] [15429.727955] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15429.767220] LNet: Added LNI 192.168.203.137@tcp46 [8/256/0/180] [15438.394883] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15438.440906] LNet: Added LNI 192.168.203.137@tcp47 [8/256/0/180] [15447.235365] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15447.269803] LNet: Added LNI 192.168.203.137@tcp48 [8/256/0/180] [15456.166668] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15456.213826] LNet: Added LNI 192.168.203.137@tcp49 [8/256/0/180] [15467.021220] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15467.075130] LNet: Added LNI 192.168.203.137@tcp50 [8/256/0/180] [15494.641920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15494.651346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15494.671882] Lustre: Skipped 1 previous similar message [15494.704020] Lustre: Skipped 2 previous similar messages [15499.113819] Lustre: server umount lustre-MDT0000 complete [15504.865572] LustreError: 473136:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15504.903534] LustreError: 473136:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [15505.238480] LustreError: 473123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789081355 with bad export cookie 6224950342070069445 [15505.242262] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15505.271827] LustreError: 473123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15506.289585] Lustre: server umount lustre-MDT0001 complete [15525.343968] Lustre: 472401:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789081359/real 1789081359] req@000000005ba33dcd x1875987316004864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789081375 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15525.357235] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15525.364249] Lustre: Skipped 1 previous similar message [15526.438458] Lustre: 472401:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789081360/real 1789081360] req@000000008aaf6485 x1875987316004992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789081376 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15529.444556] Lustre: 472399:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789081363/real 1789081363] req@00000000930ac39d x1875987316005120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789081379 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15530.805744] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [15531.551426] Lustre: 472399:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789081365/real 1789081365] req@000000002206a17e x1875987316005376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789081381 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15537.631183] Lustre: 472402:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789081371/real 1789081371] req@000000005fac0666 x1875987316005504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789081387 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15540.397035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15551.007487] LustreError: 472398:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000008ab59e x1875987316006144/t0(0) o250->MGC192.168.203.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15551.381449] LustreError: 475094:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15551.453911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15556.036054] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [15556.595169] LustreError: 475095:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15559.657697] LustreError: 475094:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15559.668583] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15564.775514] LustreError: 475095:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15565.723243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15566.219292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15570.356357] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15571.316194] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [15579.441869] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15585.933761] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15607.272525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15607.283916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15607.292315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15607.776393] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15609.317655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15609.332089] Lustre: Skipped 2 previous similar messages [15611.988496] Lustre: server umount lustre-OST0000 complete [15617.505154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15617.523807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15617.533498] Lustre: Skipped 1 previous similar message [15617.537346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15624.680069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15624.690196] Lustre: Skipped 3 previous similar messages [15630.303225] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15630.699611] Lustre: server umount lustre-MDT0000 complete [15634.913441] LustreError: 499382:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15634.931583] LustreError: 499382:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15635.201826] LustreError: 483759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789081485 with bad export cookie 6224950342070071979 [15635.208131] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15635.224031] LustreError: 483759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15635.701393] Lustre: server umount lustre-MDT0001 complete [15647.532304] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [15650.229319] Key type lgssc unregistered [15650.665938] LNet: 502187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15650.675657] LNetError: 502187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15650.691140] LNet: Removed LNI 192.168.203.137@tcp [15650.700661] LNet: Removed LNI 192.168.203.137@tcp10 [15650.706485] LNet: Removed LNI 192.168.203.137@tcp11 [15650.713242] LNet: Removed LNI 192.168.203.137@tcp12 [15650.724742] LNet: Removed LNI 192.168.203.137@tcp13 [15650.733842] LNet: Removed LNI 192.168.203.137@tcp14 [15650.740659] LNet: Removed LNI 192.168.203.137@tcp15 [15650.747622] LNet: Removed LNI 192.168.203.137@tcp16 [15650.753470] LNet: Removed LNI 192.168.203.137@tcp17 [15650.760522] LNet: Removed LNI 192.168.203.137@tcp18 [15650.763572] LNet: Removed LNI 192.168.203.137@tcp19 [15650.766994] LNet: Removed LNI 192.168.203.137@tcp20 [15650.771487] LNet: Removed LNI 192.168.203.137@tcp21 [15650.775069] LNet: Removed LNI 192.168.203.137@tcp22 [15650.778546] LNet: Removed LNI 192.168.203.137@tcp23 [15650.784602] LNet: Removed LNI 192.168.203.137@tcp24 [15650.787580] LNet: Removed LNI 192.168.203.137@tcp25 [15650.793377] LNet: Removed LNI 192.168.203.137@tcp26 [15650.802454] LNet: Removed LNI 192.168.203.137@tcp27 [15650.808203] LNet: Removed LNI 192.168.203.137@tcp28 [15650.810756] LNet: Removed LNI 192.168.203.137@tcp29 [15650.812829] LNet: Removed LNI 192.168.203.137@tcp30 [15650.821163] LNet: Removed LNI 192.168.203.137@tcp31 [15650.824913] LNet: Removed LNI 192.168.203.137@tcp32 [15650.829348] LNet: Removed LNI 192.168.203.137@tcp33 [15650.836750] LNet: Removed LNI 192.168.203.137@tcp34 [15650.838812] LNet: Removed LNI 192.168.203.137@tcp35 [15650.844350] LNet: Removed LNI 192.168.203.137@tcp36 [15650.849561] LNet: Removed LNI 192.168.203.137@tcp37 [15650.857605] LNet: Removed LNI 192.168.203.137@tcp38 [15650.862391] LNet: Removed LNI 192.168.203.137@tcp39 [15650.866092] LNet: Removed LNI 192.168.203.137@tcp40 [15650.880731] LNet: Removed LNI 192.168.203.137@tcp41 [15650.883279] LNet: Removed LNI 192.168.203.137@tcp42 [15650.885751] LNet: Removed LNI 192.168.203.137@tcp43 [15650.889443] LNet: Removed LNI 192.168.203.137@tcp44 [15650.891921] LNet: Removed LNI 192.168.203.137@tcp45 [15650.898124] LNet: Removed LNI 192.168.203.137@tcp46 [15650.901866] LNet: Removed LNI 192.168.203.137@tcp47 [15650.904372] LNet: Removed LNI 192.168.203.137@tcp48 [15650.906931] LNet: Removed LNI 192.168.203.137@tcp49 [15650.915809] LNet: Removed LNI 192.168.203.137@tcp50 [15651.652791] Key type .llcrypt unregistered [15651.654713] Key type ._llcrypt unregistered [15672.585474] Key type ._llcrypt registered [15672.587877] Key type .llcrypt registered [15672.712370] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [15688.979810] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [15690.945987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15690.991088] alg: No test for adler32 (adler32-zlib) [15692.083074] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [15692.470872] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [15694.231211] Key type lgssc registered [15695.485796] Lustre: Echo OBD driver; http://www.lustre.org/ [15734.880516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15734.913462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15736.273909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15736.333652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15736.449690] Lustre: lustre-MDT0000: new disk, initializing [15736.543606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15736.566904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15741.579506] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [15753.552725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15753.708699] Lustre: 505382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15753.811517] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15753.822033] Lustre: Skipped 1 previous similar message [15753.937331] Lustre: lustre-MDT0001: new disk, initializing [15754.105496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15754.157175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15754.170579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15760.136407] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [15772.760799] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15778.664274] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15786.973065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15787.267512] Lustre: *** cfs_fail_loc=90f, val=0*** [15787.407535] Lustre: lustre-OST0000: new disk, initializing [15787.415519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15787.427760] Lustre: 507400:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15787.552245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15794.463479] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [15797.810235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15797.821972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15797.970727] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15804.160317] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15813.603382] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15813.619442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15813.663836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15818.722904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15818.731992] Lustre: Skipped 2 previous similar messages [15823.845299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15823.859919] Lustre: Skipped 1 previous similar message [15826.399126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15826.726731] Lustre: server umount lustre-OST0000 complete [15834.080943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15834.084434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15834.095180] Lustre: Skipped 2 previous similar messages [15834.110362] Lustre: Skipped 1 previous similar message [15837.622912] Lustre: server umount lustre-MDT0000 complete [15843.435422] LustreError: 505376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789081693 with bad export cookie 2188714994546847226 [15843.458258] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15843.460328] LustreError: 505376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15844.086769] Lustre: server umount lustre-MDT0001 complete [15858.802448] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [15861.494862] Key type lgssc unregistered [15862.002737] LNet: 509308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15862.018154] LNetError: 509308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15862.040339] LNet: Removed LNI 192.168.203.137@tcp [15863.320428] Key type .llcrypt unregistered [15863.323119] Key type ._llcrypt unregistered [15885.468984] Key type ._llcrypt registered [15885.471735] Key type .llcrypt registered [15885.590556] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [15887.144436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15887.220511] alg: No test for adler32 (adler32-zlib) [15888.251570] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [15888.476117] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [15890.167225] Key type lgssc registered [15891.389374] Lustre: Echo OBD driver; http://www.lustre.org/ [15898.881497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15898.907650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15900.458477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15904.349668] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [15910.897760] LustreError: 510576:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15910.913961] LustreError: 510576:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [15914.056796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15915.049855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15921.288789] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [15929.536819] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15935.497551] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15943.314146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15943.939719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15951.616343] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [15958.864328] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15968.625544] loop0: detected capacity change from 0 to 204800000 [15968.783482] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15968.800214] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15968.816958] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15974.383351] loop0: detected capacity change from 0 to 204800000 [15974.568232] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15974.580149] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15974.591840] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15979.471933] loop0: detected capacity change from 0 to 204800000 [15984.292291] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15984.437754] Lustre: fs15246-OST0000: new disk, initializing [15984.440875] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15984.446076] Lustre: 514170:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15984.491129] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15989.723950] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [15998.296093] loop1: detected capacity change from 0 to 204800000 [16004.886794] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16005.177612] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16005.273285] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16005.402383] Lustre: fs15246-MDT0000: new disk, initializing [16005.529849] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16005.559843] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16005.649375] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16006.689098] Lustre: 515292:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000013a5c3d3 x1875988172551808/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789081872 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16011.310619] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [16011.882692] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16011.900675] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16011.947213] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16037.368718] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16037.389153] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16038.914812] Lustre: server umount fs15246-MDT0000 complete [16042.472524] LustreError: 510571:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16047.124480] Lustre: server umount fs15246-OST0000 complete [16058.848549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16058.866601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16058.903993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16061.922402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16061.937487] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16061.970566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16064.864063] Lustre: server umount lustre-OST0000 complete [16073.184121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16073.215110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16073.223320] Lustre: Skipped 3 previous similar messages [16075.805867] Lustre: server umount lustre-MDT0000 complete [16078.304652] LustreError: 510577:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16078.326855] LustreError: 510577:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16079.568325] LustreError: 510556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789081929 with bad export cookie 7594077001886547097 [16079.572924] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16079.584919] LustreError: 510556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16079.977196] Lustre: server umount lustre-MDT0001 complete [16091.053992] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [16093.329161] Key type lgssc unregistered [16093.581713] LNet: 518389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16093.589233] LNetError: 518389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16093.601218] LNet: Removed LNI 192.168.203.137@tcp [16094.593160] Key type .llcrypt unregistered [16094.595119] Key type ._llcrypt unregistered [16116.456611] Key type ._llcrypt registered [16116.459675] Key type .llcrypt registered [16116.554725] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [16117.952701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16118.201425] alg: No test for adler32 (adler32-zlib) [16119.499191] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [16119.836415] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [16121.631208] Key type lgssc registered [16123.052122] Lustre: Echo OBD driver; http://www.lustre.org/ [16160.995612] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [16169.981698] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [16205.758286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16205.785130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16206.967974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16206.990738] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16207.063451] Lustre: lustre-MDT0000: new disk, initializing [16207.182658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16207.201654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16211.364657] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [16220.905075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16221.046849] Lustre: 522851:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16221.101635] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16221.105933] Lustre: Skipped 1 previous similar message [16221.170322] Lustre: lustre-MDT0001: new disk, initializing [16221.246308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16221.287192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16221.295967] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16224.878299] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [16232.911367] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16237.502974] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16243.509192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16243.858936] Lustre: lustre-OST0000: new disk, initializing [16243.865730] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16243.874832] Lustre: 524818:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16244.001493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16249.598033] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [16252.497889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16252.507440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16252.594022] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16257.951949] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16262.776317] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16263.062985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16266.786570] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16266.952235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16273.383576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16273.389126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16273.389251] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16273.392685] Lustre: Skipped 1 previous similar message [16273.418188] Lustre: Skipped 1 previous similar message [16278.506721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16278.517122] Lustre: Skipped 1 previous similar message [16283.103131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16283.274605] Lustre: server umount lustre-OST0000 complete [16287.713521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16287.727988] LustreError: Skipped 1 previous similar message [16287.733530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16287.755749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16288.738617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16293.437319] Lustre: server umount lustre-MDT0000 complete [16297.618473] LustreError: 522844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789082147 with bad export cookie 370093520483912971 [16297.622088] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16297.624123] LustreError: 522844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16297.994567] Lustre: server umount lustre-MDT0001 complete [16321.423557] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [16332.254396] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [16360.575120] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16111 sec ========= 19:16:48 (1789082208) [16362.114938] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:16:50 (1789082210) === [16364.933597] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:16:53 (1789082213) === [16405.341281] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [16408.305735] Key type lgssc unregistered [16408.731209] LNet: 531463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16408.742408] LNetError: 531463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16408.783414] LNet: Removed LNI 192.168.203.137@tcp [16409.627537] Key type .llcrypt unregistered [16409.629959] Key type ._llcrypt unregistered