[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 482386722 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003140] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008410] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010016] pid_max: default: 32768 minimum: 301 [ 0.011131] LSM: Security Framework initializing [ 0.012060] Yama: becoming mindful. [ 0.013046] SELinux: Initializing. [ 0.014077] *** VALIDATE selinux *** [ 0.022732] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027246] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030111] *** VALIDATE tmpfs *** [ 0.031436] *** VALIDATE proc *** [ 0.032226] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034255] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037032] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.040709] debug: unmapping init [mem 0xffffffffb4a59000-0xffffffffb4a60fff] [ 0.042172] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043680] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044026] ... version: 2 [ 0.045014] ... bit width: 48 [ 0.046014] ... generic registers: 4 [ 0.047014] ... value mask: 0000ffffffffffff [ 0.048016] ... max period: 00007fffffffffff [ 0.049015] ... fixed-purpose events: 3 [ 0.050013] ... event mask: 000000070000000f [ 0.051318] rcu: Hierarchical SRCU implementation. [ 0.053409] smp: Bringing up secondary CPUs ... [ 0.054587] x86: Booting SMP configuration: [ 0.055028] .... node #0, CPUs: #1 #2 #3 [ 0.059011] smp: Brought up 1 node, 4 CPUs [ 0.061017] smpboot: Max logical packages: 1 [ 0.062012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136710] node 0 deferred pages initialised in 70ms [ 0.139098] devtmpfs: initialized [ 0.140118] x86/mm: Memory block size: 128MB [ 0.142356] gcov: version magic: 0x41383552 [ 0.144216] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.146108] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.148294] pinctrl core: initialized pinctrl subsystem [ 0.150162] [ 0.150538] ************************************************************* [ 0.152012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154010] ** ** [ 0.156008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157008] ** ** [ 0.159008] ** This means that this kernel is built to expose internal ** [ 0.161008] ** IOMMU data structures, which may compromise security on ** [ 0.162007] ** your system. ** [ 0.164008] ** ** [ 0.165008] ** If you see this message and you are not debugging the ** [ 0.167010] ** kernel, report this immediately to your vendor! ** [ 0.169012] ** ** [ 0.171012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173011] ************************************************************* [ 0.175462] NET: Registered protocol family 16 [ 0.177636] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.180090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.183089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.187051] cpuidle: using governor menu [ 0.188579] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.190711] PCI: Using configuration type 1 for base access [ 0.193156] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.202051] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.203033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.204193] cryptd: max_cpu_qlen set to 1000 [ 0.207390] ACPI: Added _OSI(Module Device) [ 0.209012] ACPI: Added _OSI(Processor Device) [ 0.210010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.211013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.215392] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.220528] ACPI: Interpreter enabled [ 0.222051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.223000] ACPI: Using IOAPIC for interrupt routing [ 0.223000] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.226348] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.233487] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.235036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.237019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.239080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.243212] acpiphp: Slot [2] registered [ 0.244170] acpiphp: Slot [5] registered [ 0.246146] acpiphp: Slot [6] registered [ 0.247161] acpiphp: Slot [7] registered [ 0.249132] acpiphp: Slot [8] registered [ 0.250073] acpiphp: Slot [9] registered [ 0.251123] acpiphp: Slot [10] registered [ 0.252105] acpiphp: Slot [3] registered [ 0.253077] acpiphp: Slot [4] registered [ 0.254027] acpiphp: Slot [11] registered [ 0.255123] acpiphp: Slot [12] registered [ 0.256070] acpiphp: Slot [13] registered [ 0.257093] acpiphp: Slot [14] registered [ 0.258094] acpiphp: Slot [15] registered [ 0.259089] acpiphp: Slot [16] registered [ 0.261106] acpiphp: Slot [17] registered [ 0.262111] acpiphp: Slot [18] registered [ 0.263088] acpiphp: Slot [19] registered [ 0.264097] acpiphp: Slot [20] registered [ 0.265072] acpiphp: Slot [21] registered [ 0.266064] acpiphp: Slot [22] registered [ 0.267104] acpiphp: Slot [23] registered [ 0.269162] acpiphp: Slot [24] registered [ 0.270124] acpiphp: Slot [25] registered [ 0.271116] acpiphp: Slot [26] registered [ 0.273143] acpiphp: Slot [27] registered [ 0.274150] acpiphp: Slot [28] registered [ 0.276119] acpiphp: Slot [29] registered [ 0.278123] acpiphp: Slot [30] registered [ 0.279134] acpiphp: Slot [31] registered [ 0.280066] PCI host bridge to bus 0000:00 [ 0.282038] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.284026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.286028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.289023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.292026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.294026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.296192] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.299364] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.303000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.312015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.317504] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.320027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.322016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.324018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.326082] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.329882] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.332065] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.334859] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.340024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.354000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.358970] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.364275] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.372015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.381017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.399021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.411667] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.419014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.425015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.444017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.455025] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.462017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.470033] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.488023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.497574] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.505020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.510022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.526016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.535367] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.540022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.546024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.566041] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.575000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.582017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.590016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.606032] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.620229] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.622462] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.625449] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.627770] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.630323] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.635238] iommu: Default domain type: Passthrough [ 0.636468] SCSI subsystem initialized [ 0.637190] ACPI: bus type USB registered [ 0.638126] usbcore: registered new interface driver usbfs [ 0.640092] usbcore: registered new interface driver hub [ 0.642092] usbcore: registered new device driver usb [ 0.644196] pps_core: LinuxPPS API ver. 1 registered [ 0.646013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.649075] PTP clock support registered [ 0.652079] EDAC MC: Ver: 3.0.0 [ 0.653178] PCI: Using ACPI for IRQ routing [ 0.656000] NetLabel: Initializing [ 0.657015] NetLabel: domain hash size = 128 [ 0.658015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.660107] NetLabel: unlabeled traffic allowed by default [ 0.663109] vgaarb: loaded [ 0.664274] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.666022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.677151] clocksource: Switched to clocksource kvm-clock [ 0.790150] VFS: Disk quotas dquot_6.6.0 [ 0.791814] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.794163] *** VALIDATE ramfs *** [ 0.795380] *** VALIDATE hugetlbfs *** [ 0.796980] pnp: PnP ACPI init [ 0.799449] pnp: PnP ACPI: found 6 devices [ 0.819852] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.823346] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.825443] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.827293] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.829773] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.832042] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.834020] NET: Registered protocol family 2 [ 0.836730] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.841371] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.844850] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.851232] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.855074] TCP: Hash tables configured (established 65536 bind 65536) [ 0.858148] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.861829] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.864947] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.868493] NET: Registered protocol family 1 [ 0.871884] RPC: Registered named UNIX socket transport module. [ 0.873838] RPC: Registered udp transport module. [ 0.875426] RPC: Registered tcp transport module. [ 0.877056] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.879176] NET: Registered protocol family 44 [ 0.880596] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.882790] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.885140] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.887547] PCI: CLS 0 bytes, default 64 [ 0.889172] Unpacking initramfs... [ 2.205940] debug: unmapping init [mem 0xffff89657cc54000-0xffff89657ffbffff] [ 2.208485] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.210121] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.212257] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.712102] Initialise system trusted keyrings [ 2.713715] Key type blacklist registered [ 2.715772] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.728942] zbud: loaded [ 2.733331] *** VALIDATE nfs *** [ 2.734671] *** VALIDATE nfs4 *** [ 2.736548] pstore: using deflate compression [ 2.740102] Platform Keyring initialized [ 2.852370] NET: Registered protocol family 38 [ 2.854419] Key type asymmetric registered [ 2.856079] Asymmetric key parser 'x509' registered [ 2.858081] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.860985] io scheduler mq-deadline registered [ 2.862880] io scheduler kyber registered [ 2.864740] io scheduler bfq registered [ 2.866649] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.869274] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.872046] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.874976] ACPI: Power Button [PWRF] [ 2.880368] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.887564] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.906722] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.921261] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.950450] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.978171] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.005642] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.009382] Non-volatile memory driver v1.3 [ 3.010449] Linux agpgart interface v0.103 [ 3.036969] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.038823] vda: detected capacity change from 0 to 74702848 [ 3.050060] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.053132] vdb: detected capacity change from 0 to 1073741824 [ 3.065787] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.068113] vdc: detected capacity change from 0 to 2621440000 [ 3.079638] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.081478] vdd: detected capacity change from 0 to 2621440000 [ 3.093062] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.095312] vde: detected capacity change from 0 to 4294967296 [ 3.107985] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.110726] vdf: detected capacity change from 0 to 4294967296 [ 3.120941] libphy: Fixed MDIO Bus: probed [ 3.135228] usbcore: registered new interface driver usbserial_generic [ 3.137173] usbserial: USB Serial support registered for generic [ 3.139127] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.143566] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.145093] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.147568] mousedev: PS/2 mouse device common for all mice [ 3.151603] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.152333] rtc_cmos 00:05: RTC can wake from S4 [ 3.159409] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.164233] rtc_cmos 00:05: registered as rtc0 [ 3.172578] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.176089] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.177098] intel_pstate: CPU model not supported [ 3.183366] hid: raw HID events driver (C) Jiri Kosina [ 3.185406] usbcore: registered new interface driver usbhid [ 3.187163] usbhid: USB HID core driver [ 3.188795] drop_monitor: Initializing network drop monitor service [ 3.190929] Initializing XFRM netlink socket [ 3.192977] NET: Registered protocol family 10 [ 3.195709] Segment Routing with IPv6 [ 3.197084] NET: Registered protocol family 17 [ 3.198911] mpls_gso: MPLS GSO support [ 3.205146] RAS: Correctable Errors collector initialized. [ 3.207372] AVX version of gcm_enc/dec engaged. [ 3.208887] AES CTR mode by8 optimization enabled [ 3.291789] sched_clock: Marking stable (3291723186, 0)->(4190442540, -898719354) [ 3.295370] registered taskstats version 1 [ 3.297266] Loading compiled-in X.509 certificates [ 3.299445] zswap: loaded using pool lzo/zbud [ 3.323945] Key type big_key registered [ 3.339878] Key type encrypted registered [ 3.341300] ima: No TPM chip found, activating TPM-bypass! [ 3.343148] ima: Allocated hash algorithm: sha1 [ 3.344747] ima: No architecture policies found [ 3.346214] evm: Initialising EVM extended attributes: [ 3.347763] evm: security.selinux [ 3.348836] evm: security.ima [ 3.349681] evm: security.capability [ 3.350705] evm: HMAC attrs: 0x1 [ 3.353104] rtc_cmos 00:05: setting system clock to 2026-08-17 13:55:45 UTC (1786974945) [ 3.359123] debug: unmapping init [mem 0xffffffffb5a03000-0xffffffffb5bfffff] [ 3.361944] debug: unmapping init [mem 0xffffffffb4782000-0xffffffffb4a58fff] [ 3.368213] Write protecting the kernel read-only data: 28672k [ 3.371743] debug: unmapping init [mem 0xffffffffb2e03000-0xffffffffb2ffffff] [ 3.374832] debug: unmapping init [mem 0xffffffffb3714000-0xffffffffb37fffff] [ 3.416647] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.424498] systemd[1]: Detected virtualization kvm. [ 3.426335] systemd[1]: Detected architecture x86-64. [ 3.428051] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.454153] systemd[1]: No hostname configured. [ 3.455888] systemd[1]: Set hostname to . [ 3.457996] random: systemd: uninitialized urandom read (16 bytes read) [ 3.460458] systemd[1]: Initializing machine ID from random generator. [ 3.603167] random: systemd: uninitialized urandom read (16 bytes read) [ 3.606224] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.611674] random: systemd: uninitialized urandom read (16 bytes read) [ 3.614600] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.619391] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.262285] device-mapper: uevent: version 1.0.3 [ 4.263672] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 4.800612] random: fast init done [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.972981] virtio_net virtio0 ens2: renamed from eth0 [ 4.994297] scsi host0: ata_piix [ 5.015729] scsi host1: ata_piix [ 5.017381] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.020087] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.873528] random: crng init done [ 9.876363] random: 7 urandom warning(s) missed due to ratelimiting [ 9.886723] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ 11.307592] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.545146] printk: systemd: 25 output lines suppressed due to ratelimiting [ 14.367668] SELinux: Disabled at runtime. [ 14.549078] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.562453] systemd[1]: Detected virtualization kvm. [ 14.564597] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.267722] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.274472] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.287784] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.291821] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.304466] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.333644] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.357599] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ 16.732961] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 17.531134] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.633206] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.696595] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.314095] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.525568] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is runni[ 24.428333] hrtimer: interrupt took 6421707 ns ng for Configur…-only root support (8s / no limit)[ 24.526652] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (8s / no limit)[ 25.256667] NFS: Registering the id_resolver key type [ 25.259667] Key type id_resolver registered [ 25.261954] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg237-server login: [ 82.385427] libcfs: loading out-of-tree module taints kernel. [ 82.429478] Key type ._llcrypt registered [ 82.431681] Key type .llcrypt registered [ 82.528822] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 102.574762] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 104.587431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 104.599712] alg: No test for adler32 (adler32-zlib) [ 106.372427] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 107.610817] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 109.671512] Key type lgssc registered [ 111.902771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 132.373380] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 181.225764] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 196.380674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 196.435372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 197.786667] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 197.833222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 197.900501] Lustre: lustre-MDT0000: new disk, initializing [ 197.985174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 198.005949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 202.816390] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.369869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 217.497528] Lustre: 6518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 217.531625] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 217.535666] Lustre: Skipped 1 previous similar message [ 217.636775] Lustre: lustre-MDT0001: new disk, initializing [ 217.703379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.752240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 217.770790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 224.180517] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.154060] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 239.990544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.434582] Lustre: lustre-OST0000: new disk, initializing [ 240.442704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 240.452467] Lustre: 8455:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 240.562233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 246.313485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 246.330170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 246.395955] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 247.042901] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 262.292874] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 262.429649] Lustre: lustre-OST0001: new disk, initializing [ 262.436719] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 262.441794] Lustre: 9529:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 262.493110] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 268.836426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 268.859144] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 268.918261] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 269.376567] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 282.780717] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 291.617632] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 299.109240] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing check_logdir /tmp/testlogs/ [ 305.146403] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing yml_node [ 310.033203] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 312.714961] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 315.710276] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 317.605627] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 17 10:00:58 EDT 2026 [ 341.025453] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 343.676433] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 360.928629] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.932411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 360.953075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 360.958769] Lustre: Skipped 3 previous similar messages [ 360.980559] Lustre: Skipped 2 previous similar messages [ 366.121177] Lustre: server umount lustre-MDT0000 complete [ 370.686374] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786975312 with bad export cookie 9125370901460893516 [ 370.691881] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 370.694968] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 371.172300] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 371.173627] LustreError: 6526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 371.203861] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 371.237881] LustreError: 6526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 371.261913] Lustre: Skipped 2 previous similar messages [ 375.200084] LustreError: 6524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 375.201507] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 375.248829] Lustre: Skipped 1 previous similar message [ 377.101294] Lustre: server umount lustre-MDT0001 complete [ 381.241290] Lustre: server umount lustre-OST0000 complete [ 385.189453] Lustre: server umount lustre-OST0001 complete [ 392.583909] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 401.787710] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 441.092037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 441.350547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 441.378309] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 441.488763] Lustre: lustre-MDT0000: new disk, initializing [ 441.661045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 441.686924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 446.965483] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 459.405393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 459.643181] Lustre: 15097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 459.713199] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 459.722198] Lustre: Skipped 1 previous similar message [ 459.877980] Lustre: lustre-MDT0001: new disk, initializing [ 459.986732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 460.048496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 460.060990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 466.266454] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 477.683765] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 485.453925] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 494.285821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 494.640974] Lustre: lustre-OST0000: new disk, initializing [ 494.645350] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 494.656443] Lustre: 17067:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 494.766987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 495.954617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 495.971385] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 496.169183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 501.225137] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 512.866477] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 519.371371] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 519.741606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 526.609688] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 526.905037] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 531.940777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 531.944206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.952012] Lustre: Skipped 1 previous similar message [ 531.957959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 531.971076] Lustre: Skipped 1 previous similar message [ 535.448492] Lustre: server umount lustre-OST0000 complete [ 542.183649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 542.205488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 542.207470] LustreError: Skipped 1 previous similar message [ 542.208208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 542.235487] Lustre: Skipped 2 previous similar messages [ 546.419986] Lustre: server umount lustre-MDT0000 complete [ 547.299188] LustreError: 15105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 547.331860] LustreError: 15105:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 550.236744] LustreError: 17066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786975492 with bad export cookie 9125370901460895049 [ 550.240737] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 550.247797] LustreError: 17066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 550.655564] Lustre: server umount lustre-MDT0001 complete [ 554.873650] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:04:55 (1786975495) [ 557.338908] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 559.134300] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:04:59 (1786975499) [ 567.245244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.231342] LustreError: 19479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.374194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 573.423816] LustreError: 19480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 575.843184] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 583.652385] LustreError: 19479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.687381] LustreError: 19479:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 588.635597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 594.510981] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 603.583457] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 609.143963] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 616.002569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 622.781886] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 630.156393] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 649.188531] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 649.204089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 649.204894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 649.213961] Lustre: Skipped 1 previous similar message [ 653.963734] Lustre: server umount lustre-OST0000 complete [ 659.426504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 659.447842] Lustre: Skipped 2 previous similar messages [ 664.405648] Lustre: server umount lustre-MDT0000 complete [ 664.551185] LustreError: 19479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 664.584994] LustreError: 19479:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 668.863302] LustreError: 19461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786975611 with bad export cookie 9125370901460895931 [ 668.864435] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 668.874568] LustreError: 19461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 669.258532] Lustre: server umount lustre-MDT0001 complete [ 680.874750] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 683.576939] Key type lgssc unregistered [ 683.806755] LNet: 23038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.810479] LNetError: 23038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 683.824972] LNet: Removed LNI 192.168.202.137@tcp [ 684.509157] Key type .llcrypt unregistered [ 684.511134] Key type ._llcrypt unregistered [ 704.758430] Key type ._llcrypt registered [ 704.760650] Key type .llcrypt registered [ 704.859252] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 705.608367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 705.789264] alg: No test for adler32 (adler32-zlib) [ 706.810146] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 707.016525] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 708.687155] Key type lgssc registered [ 709.801603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 717.752217] loop: module loaded [ 717.834272] loop0: detected capacity change from 0 to 204800000 [ 718.076777] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 718.088632] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 718.352539] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 724.209887] loop0: detected capacity change from 0 to 204800000 [ 724.542043] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 724.566110] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 724.578316] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 729.132272] loop0: detected capacity change from 0 to 204800000 [ 734.881140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 734.913701] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 736.238487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 736.294117] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 736.385395] Lustre: lustre-MDT0000: new disk, initializing [ 736.469689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 736.489569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 741.323967] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 750.304706] loop1: detected capacity change from 0 to 204800000 [ 756.873739] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 757.338175] Lustre: lustre-OST07c6: new disk, initializing [ 757.348554] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 757.355686] Lustre: Skipped 1 previous similar message [ 757.365800] Lustre: 26202:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 757.469720] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 764.693858] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 765.581965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 765.592570] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 765.630624] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 769.867943] Lustre: 25190:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 769.879447] Lustre: 25190:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 769.888798] Lustre: 25190:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 769.895269] Lustre: 25190:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 769.905025] Lustre: 25190:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 769.925707] Lustre: 25190:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 785.878309] Lustre: 25190:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 785.896181] Lustre: 25190:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 785.910742] Lustre: 25190:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 785.924704] Lustre: 25190:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 785.931272] Lustre: 25190:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 785.937780] Lustre: 25190:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 790.758029] Lustre: 26830:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 790.776754] Lustre: 26830:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 790.791027] Lustre: 26830:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 790.808370] Lustre: 26830:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.815600] Lustre: 26830:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 790.833043] Lustre: 26830:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.843034] Lustre: 26830:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 790.854347] Lustre: 26830:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.866451] Lustre: 26830:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 790.874739] Lustre: 26830:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.891516] Lustre: 26830:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 790.903513] Lustre: 26830:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.152600] Lustre: 25191:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 794.166295] Lustre: 25191:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 794.176936] Lustre: 25191:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 794.196071] Lustre: 25191:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.207155] Lustre: 25191:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 794.218167] Lustre: 25191:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.231407] Lustre: 25191:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 794.240765] Lustre: 25191:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.250915] Lustre: 25191:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 794.266412] Lustre: 25191:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.275959] Lustre: 25191:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 794.286679] Lustre: 25191:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.492820] Lustre: 26830:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 799.513940] Lustre: 26830:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 799.522200] Lustre: 26830:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 799.528243] Lustre: 26830:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.533441] Lustre: 26830:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 799.539883] Lustre: 26830:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.545794] Lustre: 26830:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 799.552246] Lustre: 26830:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.557141] Lustre: 26830:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 799.561400] Lustre: 26830:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.566756] Lustre: 26830:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 799.572965] Lustre: 26830:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 807.505355] Lustre: 25191:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 807.516494] Lustre: 25191:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 807.522682] Lustre: 25191:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 807.529069] Lustre: 25191:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 4 previous similar messages [ 807.539361] Lustre: 25191:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 807.547744] Lustre: 25191:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 4 previous similar messages [ 807.558315] Lustre: 25191:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 807.565190] Lustre: 25191:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 4 previous similar messages [ 807.576263] Lustre: 25191:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/3, delete: 0/0/0 [ 807.588243] Lustre: 25191:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 4 previous similar messages [ 807.601206] Lustre: 25191:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 807.615987] Lustre: 25191:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 4 previous similar messages [ 863.219134] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 863.229535] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 863.229914] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 866.435247] Lustre: server umount lustre-OST07c6 complete [ 877.343774] Lustre: server umount lustre-MDT0000 complete [ 895.319199] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 898.293035] Key type lgssc unregistered [ 898.668706] LNet: 28774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.682213] LNetError: 28774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.701594] LNet: Removed LNI 192.168.202.137@tcp [ 899.568989] Key type .llcrypt unregistered [ 899.571186] Key type ._llcrypt unregistered [ 920.631830] Key type ._llcrypt registered [ 920.641435] Key type .llcrypt registered [ 920.780723] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 922.035231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 922.096538] alg: No test for adler32 (adler32-zlib) [ 923.099897] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 923.361131] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 925.007587] Key type lgssc registered [ 926.268593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 934.470947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 934.489614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 936.049513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 940.650391] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 946.657680] LustreError: 30046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 946.682471] LustreError: 30046:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 949.481357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 949.912753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 954.485916] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 963.961317] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 973.127550] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 981.981361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 982.532714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 989.642959] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 998.198623] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1006.087243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 1026.533745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1026.536822] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1026.586988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1027.893630] Lustre: server umount lustre-OST0000 complete [ 1036.769788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1036.773734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1036.777644] Lustre: Skipped 1 previous similar message [ 1036.778570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1036.813286] Lustre: Skipped 2 previous similar messages [ 1039.222903] Lustre: server umount lustre-MDT0000 complete [ 1041.896333] LustreError: 30774:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1041.924830] LustreError: 30774:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1043.676426] LustreError: 31892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786975985 with bad export cookie 1943237156970606930 [ 1043.681190] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1043.690039] LustreError: 31892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1044.057733] Lustre: server umount lustre-MDT0001 complete [ 1056.706878] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1060.241932] Key type lgssc unregistered [ 1060.583882] LNet: 33809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.592493] LNetError: 33809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1061.670826] LNet: Removed LNI 192.168.202.137@tcp [ 1062.596595] Key type .llcrypt unregistered [ 1062.599211] Key type ._llcrypt unregistered [ 1082.049496] Key type ._llcrypt registered [ 1082.052949] Key type .llcrypt registered [ 1082.139673] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1083.162912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1083.280923] alg: No test for adler32 (adler32-zlib) [ 1084.408952] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 1084.671125] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1086.335203] Key type lgssc registered [ 1087.336843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1096.168413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1096.215988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1097.804536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1102.956833] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1106.914825] LustreError: 35082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1106.924273] LustreError: 35082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1111.897084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1112.035465] LustreError: 35081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1112.193896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1117.075400] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1127.000532] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1133.167453] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1140.496192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1140.836077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1141.881166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1147.434563] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1155.688545] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1167.841735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1167.844740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1167.850192] LustreError: Skipped 1 previous similar message [ 1167.858501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1167.879530] Lustre: Skipped 1 previous similar message [ 1171.506533] Lustre: server umount lustre-OST0000 complete [ 1178.089485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1178.107199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1178.111368] Lustre: Skipped 2 previous similar messages [ 1181.798765] Lustre: server umount lustre-MDT0000 complete [ 1183.202106] LustreError: 35082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1183.219589] LustreError: 35082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1185.928121] LustreError: 35818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786976128 with bad export cookie 8082223396995685128 [ 1185.935922] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1185.937603] LustreError: 35818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1186.390339] Lustre: server umount lustre-MDT0001 complete [ 1199.190879] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1202.137944] Key type lgssc unregistered [ 1202.518276] LNet: 38619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1202.521756] LNetError: 38619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1202.543167] LNet: Removed LNI 192.168.202.137@tcp [ 1203.537960] Key type .llcrypt unregistered [ 1203.539757] Key type ._llcrypt unregistered [ 1231.118653] Key type ._llcrypt registered [ 1231.121461] Key type .llcrypt registered [ 1231.247238] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1232.590784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1232.652570] alg: No test for adler32 (adler32-zlib) [ 1233.754776] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 1234.032266] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1235.687480] Key type lgssc registered [ 1237.063373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1245.228328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1245.260525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1246.800149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1252.030896] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1255.914131] LustreError: 40037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1255.928405] LustreError: 40037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1261.025410] LustreError: 40038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1262.726650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1263.254526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1268.004658] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1276.525129] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1282.998721] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1290.706509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1291.168569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1292.226476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1297.468447] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1305.306464] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1318.342725] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.37@tcp (stopping) [ 1318.887303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1318.905718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1318.907550] Lustre: Skipped 1 previous similar message [ 1318.922608] Lustre: Skipped 2 previous similar messages [ 1321.646679] Lustre: server umount lustre-MDT0000 complete [ 1323.443224] LustreError: 40032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1325.206889] LustreError: 40019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786976267 with bad export cookie 13931274090559304184 [ 1325.208176] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1325.213405] LustreError: 40019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1328.544707] LustreError: 40034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.566947] LustreError: 40034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1328.598494] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.37@tcp (stopping) [ 1329.121289] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1329.131175] Lustre: Skipped 1 previous similar message [ 1331.762510] Lustre: server umount lustre-MDT0001 complete [ 1348.162319] Lustre: server umount lustre-OST0000 complete [ 1367.829640] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1370.523784] Key type lgssc unregistered [ 1370.815822] LNet: 43904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1370.823139] LNetError: 43904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1370.846381] LNet: Removed LNI 192.168.202.137@tcp [ 1371.846830] Key type .llcrypt unregistered [ 1371.849280] Key type ._llcrypt unregistered [ 1398.365314] Key type ._llcrypt registered [ 1398.368506] Key type .llcrypt registered [ 1398.517487] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1399.841961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1399.950961] alg: No test for adler32 (adler32-zlib) [ 1401.040553] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 1401.338591] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1403.047118] Key type lgssc registered [ 1404.119918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1411.308415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1411.330826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1412.814847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1417.068079] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1422.307198] LustreError: 45333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1422.317425] LustreError: 45333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1425.500509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1425.822439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1430.258614] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1437.351431] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1443.325916] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1450.922133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1451.354747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1457.534893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1457.821808] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1466.451646] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1478.115672] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1478.120968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1478.151798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1479.099276] Lustre: lustre-OST0000: Not available for connect from 192.168.202.37@tcp (stopping) [ 1481.185064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1481.202015] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1481.207919] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1483.176621] Lustre: server umount lustre-OST0000 complete [ 1501.665849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1501.680245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1501.687111] Lustre: Skipped 1 previous similar message [ 1507.155659] Lustre: server umount lustre-MDT0000 complete [ 1507.816136] LustreError: 45334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1507.836232] LustreError: 45334:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1510.484482] LustreError: 46069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786976452 with bad export cookie 12253298061195633199 [ 1510.488958] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1510.497628] LustreError: 46069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1516.885994] Lustre: server umount lustre-MDT0001 complete [ 1528.966903] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1531.583728] Key type lgssc unregistered [ 1531.849538] LNet: 49042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1531.858263] LNetError: 49042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1532.907923] LNet: Removed LNI 192.168.202.137@tcp [ 1533.670692] Key type .llcrypt unregistered [ 1533.675263] Key type ._llcrypt unregistered [ 1558.686687] Key type ._llcrypt registered [ 1558.692127] Key type .llcrypt registered [ 1558.787372] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1559.804641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1560.044199] alg: No test for adler32 (adler32-zlib) [ 1561.196646] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 1561.419304] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1563.143552] Key type lgssc registered [ 1564.208418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1571.914195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1571.944522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1573.530273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1577.415153] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1583.072617] LustreError: 50459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1583.086602] LustreError: 50459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1585.294575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1585.605312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1589.994323] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1597.593758] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1603.634475] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1612.024516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1612.596169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1618.768545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1620.814433] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1629.821545] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1635.533356] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1639.166646] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1651.857857] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-1710h23m11s [ 1654.041435] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1656.800907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1656.809952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1656.823124] Lustre: Skipped 2 previous similar messages [ 1656.840953] Lustre: Skipped 2 previous similar messages [ 1657.889782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1657.898848] Lustre: Skipped 3 previous similar messages [ 1662.720876] Lustre: server umount lustre-MDT0000 complete [ 1663.396269] LustreError: 52240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1666.704829] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1669.545811] LustreError: 50455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1669.584768] LustreError: 50455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1674.320373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1674.455981] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1674.689903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1674.760692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1675.701396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1679.417815] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1682.475210] Lustre: DEBUG MARKER: Wait for df (36714) ... [ 1685.072746] Lustre: DEBUG MARKER: done [ 1690.081844] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1690.104338] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1690.136521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1690.151356] Lustre: Skipped 1 previous similar message [ 1691.118759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1696.050427] Lustre: server umount lustre-OST0000 complete [ 1701.344483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1701.349399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1701.356181] Lustre: Skipped 1 previous similar message [ 1701.368296] Lustre: Skipped 2 previous similar messages [ 1706.118672] Lustre: server umount lustre-MDT0000 complete [ 1708.513665] LustreError: 53149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1708.521845] LustreError: 53149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1709.805830] LustreError: 50439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786976651 with bad export cookie 10820795615102380927 [ 1709.812875] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1709.819566] LustreError: 50439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1710.109049] Lustre: server umount lustre-MDT0001 complete [ 1719.967701] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1722.180441] Key type lgssc unregistered [ 1722.577085] LNet: 55392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.588252] LNetError: 55392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1722.600979] LNet: Removed LNI 192.168.202.137@tcp [ 1723.572207] Key type .llcrypt unregistered [ 1723.574197] Key type ._llcrypt unregistered [ 1762.977794] Key type ._llcrypt registered [ 1762.979769] Key type .llcrypt registered [ 1763.061794] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1764.329838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1764.357255] alg: No test for adler32 (adler32-zlib) [ 1765.335765] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 1765.532487] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1767.175153] Key type lgssc registered [ 1768.071128] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1776.344392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1776.388663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1777.682675] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1777.709329] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1777.885524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1782.829915] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1794.336602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1794.471035] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1794.491543] Lustre: 57599:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1794.807150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1800.656686] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1809.845699] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1816.834480] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1824.206261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1824.437364] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1824.628757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1830.304748] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1832.152027] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1836.894942] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1843.051897] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-1710h26m23s [ 1845.133125] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1852.974226] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1855.401879] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1855.554923] LustreError: 58384:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1875.935338] Lustre: lustre-MDT0000: Client f7502607-0935-4e52-a3c1-d28918d52b21 (at 192.168.202.37@tcp) reconnecting [ 1895.591142] LustreError: 58384:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1898.162349] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1900.702231] Lustre: DEBUG MARKER: done [ 1909.233240] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1909.234211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1909.243861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1909.270921] Lustre: Skipped 1 previous similar message [ 1912.453616] Lustre: server umount lustre-OST0000 complete [ 1917.922119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1917.930505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1917.960852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1917.963428] Lustre: Skipped 1 previous similar message [ 1919.022435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1919.027910] Lustre: Skipped 2 previous similar messages [ 1922.504854] Lustre: server umount lustre-MDT0000 complete [ 1926.778075] LustreError: 60887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786976868 with bad export cookie 3561409318706547574 [ 1926.784491] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1926.786937] LustreError: 60887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1927.364992] Lustre: server umount lustre-MDT0001 complete [ 1938.958487] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1941.780432] Key type lgssc unregistered [ 1941.999741] LNet: 61932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1942.004372] LNetError: 61932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1943.023574] LNet: Removed LNI 192.168.202.137@tcp [ 1943.693618] Key type .llcrypt unregistered [ 1943.701130] Key type ._llcrypt unregistered [ 1985.537191] Key type ._llcrypt registered [ 1985.539384] Key type .llcrypt registered [ 1985.685892] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1986.734953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1986.957581] alg: No test for adler32 (adler32-zlib) [ 1988.027625] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 1988.280935] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1989.952237] Key type lgssc registered [ 1991.834794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1995.499712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1995.545994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1996.917295] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1996.935302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1997.116316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1998.277806] Lustre: Failing over lustre-MDT0000 [ 1998.650926] Lustre: server umount lustre-MDT0000 complete [ 2005.637822] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:29:06 (1786976946) [ 2012.455084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.895763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2017.566497] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2027.780371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2027.844666] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2027.877654] Lustre: 64733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2028.077496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2033.456808] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2041.626996] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2048.091124] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2054.940894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2055.179393] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2055.539222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2060.903899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:291 to 0x280000401:321) [ 2061.728187] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2071.236581] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2077.928520] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2086.373044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2086.385030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2086.389853] Lustre: Skipped 1 previous similar message [ 2090.961230] Lustre: server umount lustre-MDT0000 complete [ 2094.943141] LustreError: 64724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786977037 with bad export cookie 9370672235747734853 [ 2094.945556] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2094.958068] LustreError: 64724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2095.456819] Lustre: server umount lustre-MDT0001 complete [ 2099.190558] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2104.820276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2111.976488] Lustre: 63655:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786977038/real 1786977038] req@00000000d18145cb x1873781121394176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786977054 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2112.010565] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2112.023296] Lustre: Skipped 1 previous similar message [ 2114.270643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.040365] Lustre: 63657:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786977041/real 1786977041] req@0000000020a28c81 x1873781121394304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786977057 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2117.151584] Lustre: 63657:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786977043/real 1786977043] req@00000000f83fa94a x1873781121394432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786977059 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2120.225662] LustreError: 63653:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000086b8a09a x1873781121395200/t0(0) o250->MGC192.168.202.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2120.436944] LustreError: 66581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2120.453871] LustreError: 66581:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2120.507276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2120.587478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2121.503112] Lustre: 63655:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786977047/real 1786977047] req@000000009f504984 x1873781121394688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786977063 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2121.506136] LustreError: 66582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2122.530863] LustreError: 68501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2122.570288] LustreError: 68501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2126.155875] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2127.839594] Lustre: 63655:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786977053/real 1786977053] req@00000000093b5c32 x1873781121394816/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786977069 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2127.865307] LustreError: 68503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2127.878614] LustreError: 68503:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2132.961850] LustreError: 68501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2132.975106] LustreError: 68501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2134.563229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2135.023838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2139.685232] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2140.163120] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2141.245972] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2148.941311] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2156.390660] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2165.734144] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2165.745544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2165.760676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2165.766220] Lustre: Skipped 2 previous similar messages [ 2166.762092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2166.777519] Lustre: Skipped 2 previous similar messages [ 2171.895112] LustreError: 68503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2171.916997] LustreError: 68503:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2171.973453] Lustre: server umount lustre-MDT0000 complete [ 2175.520415] LustreError: 64723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786977117 with bad export cookie 9370672235747737135 [ 2175.523488] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2175.544913] LustreError: 64723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2175.881749] Lustre: server umount lustre-MDT0001 complete [ 2181.829626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2192.178068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.375585] Lustre: 63656:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786977119/real 1786977119] req@0000000024fdc12b x1873781121425024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786977135 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2193.411541] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2193.427304] Lustre: Skipped 2 previous similar messages [ 2201.124369] LustreError: 66582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2201.225830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2201.308148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2206.302514] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2216.693686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2222.371035] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2222.792592] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2230.792772] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2236.927974] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2248.159897] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2248.167268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2248.175813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2253.284670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2253.293566] Lustre: Skipped 4 previous similar messages [ 2258.400309] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2258.571961] Lustre: server umount lustre-MDT0000 complete [ 2263.080585] LustreError: 64723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786977205 with bad export cookie 9370672235747738514 [ 2263.093157] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2263.103553] LustreError: 64723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2263.520690] LustreError: 71186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2263.524122] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2263.530070] LustreError: 71186:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2263.559203] Lustre: Skipped 2 previous similar messages [ 2263.680691] Lustre: server umount lustre-MDT0001 complete [ 2271.310187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2275.161340] Lustre: DEBUG MARKER: files compared the same [ 2277.336372] Lustre: server umount lustre-OST0000 complete [ 2294.074804] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2297.163774] Key type lgssc unregistered [ 2297.553782] LNet: 74470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2297.561428] LNetError: 74470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2297.585294] LNet: Removed LNI 192.168.202.137@tcp [ 2298.610952] Key type .llcrypt unregistered [ 2298.612825] Key type ._llcrypt unregistered [ 2325.658356] Key type ._llcrypt registered [ 2325.660641] Key type .llcrypt registered [ 2325.763953] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2326.976051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2327.024989] alg: No test for adler32 (adler32-zlib) [ 2328.162227] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 2328.410078] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2330.080787] Key type lgssc registered [ 2331.210404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2339.191255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2339.207690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2340.606971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2344.872618] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug +malloc all [ 2350.056481] LustreError: 75889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2350.068846] LustreError: 75889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2353.183783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.420246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2357.754488] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug +malloc all [ 2365.781840] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2371.235450] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2377.491656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2377.721301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2383.767688] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug +malloc all [ 2384.878375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2391.233811] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2400.225363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2400.231887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2400.234436] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2400.249600] Lustre: Skipped 1 previous similar message [ 2400.272272] Lustre: Skipped 1 previous similar message [ 2405.344597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2405.711311] Lustre: server umount lustre-OST0000 complete [ 2409.954676] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2409.964397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2409.981134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2409.988487] Lustre: Skipped 1 previous similar message [ 2415.586337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2415.593509] Lustre: Skipped 3 previous similar messages [ 2415.928079] Lustre: server umount lustre-MDT0000 complete [ 2419.739247] LustreError: 75871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786977361 with bad export cookie 10125081683450480868 [ 2419.751729] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2419.753094] LustreError: 75871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2420.055662] Lustre: server umount lustre-MDT0001 complete [ 2432.176342] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2434.840527] Key type lgssc unregistered [ 2435.137543] LNet: 79423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2435.142179] LNetError: 79423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2435.164334] LNet: Removed LNI 192.168.202.137@tcp [ 2436.044618] Key type .llcrypt unregistered [ 2436.052397] Key type ._llcrypt unregistered [ 2461.600838] Key type ._llcrypt registered [ 2461.605022] Key type .llcrypt registered [ 2461.798527] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2463.365289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2463.663749] alg: No test for adler32 (adler32-zlib) [ 2464.945398] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 2465.397140] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2467.239248] Key type lgssc registered [ 2469.211254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2479.907489] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2479.922652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2495.456526] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2500.577379] LustreError: 80814:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.137@tcp: failed processing log, type 4: rc = -110 [ 2527.268626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2534.101971] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2542.679872] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2551.297141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2551.493507] Lustre: *** cfs_fail_loc=706, val=0*** [ 2552.014509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2552.092614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2557.168265] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2557.411245] LustreError: 80839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2557.460903] LustreError: 80839:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2562.533528] LustreError: 80840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2562.553247] LustreError: 80840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2566.660775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2567.138549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2572.283675] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2580.744342] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2586.913080] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2592.743811] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2592.746107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.770772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2592.791107] Lustre: Skipped 1 previous similar message [ 2592.803467] Lustre: Skipped 1 previous similar message [ 2595.752199] Lustre: server umount lustre-OST0000 complete [ 2602.976672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2602.978871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2602.984459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2603.007467] Lustre: Skipped 1 previous similar message [ 2605.934114] Lustre: server umount lustre-MDT0000 complete [ 2608.120852] LustreError: 81901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2608.150611] LustreError: 81901:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2609.940346] LustreError: 80820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786977552 with bad export cookie 4110866898130635417 [ 2609.958591] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2609.961591] LustreError: 80820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2610.310646] Lustre: server umount lustre-MDT0001 complete [ 2623.478547] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2626.770429] Key type lgssc unregistered [ 2627.022697] LNet: 84436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2627.042370] LNetError: 84436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2627.075521] LNet: Removed LNI 192.168.202.137@tcp [ 2628.327383] Key type .llcrypt unregistered [ 2628.329575] Key type ._llcrypt unregistered [ 2656.804992] Key type ._llcrypt registered [ 2656.807591] Key type .llcrypt registered [ 2656.925845] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2658.461347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2658.492489] alg: No test for adler32 (adler32-zlib) [ 2659.656247] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 2659.918471] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2661.583522] Key type lgssc registered [ 2662.885162] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2671.347472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2671.385056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2677.535188] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2686.102332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2686.709914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2691.607506] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2692.070476] LustreError: 86541:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2692.087379] LustreError: 86541:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2696.160827] LustreError: 86544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2696.184650] LustreError: 86544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2699.601763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2699.941886] LustreError: 86545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2699.966970] LustreError: 86545:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2699.986922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2705.379861] LustreError: 86544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.397778] LustreError: 86544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2706.837739] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2710.501620] LustreError: 86540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.513239] LustreError: 86540:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2716.337857] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2716.365530] Lustre: MGS: Client 3827f1bb-0433-43a5-8808-866c18571a9b (at 0@lo) reconnecting [ 2716.394236] LustreError: 86555:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2716.642671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2716.646918] Lustre: Skipped 1 previous similar message [ 2716.815193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2716.931062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2721.797642] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2737.124955] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2737.133313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2737.150827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2737.157987] Lustre: Skipped 1 previous similar message [ 2737.634709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2742.286736] Lustre: server umount lustre-OST0000 complete [ 2747.874773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2747.882281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2747.883166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2747.883173] Lustre: Skipped 1 previous similar message [ 2747.893290] Lustre: Skipped 1 previous similar message [ 2751.890968] Lustre: server umount lustre-MDT0000 complete [ 2753.005266] LustreError: 86545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2753.022367] LustreError: 86545:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2756.261132] Lustre: server umount lustre-MDT0001 complete [ 2760.442996] Lustre: server umount MGS complete [ 2773.940937] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:41:54 (1786977714) [ 2808.809639] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 2819.413770] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2858.561981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2863.392937] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2872.303553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2872.584311] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2872.629730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2872.719146] Lustre: lustre-MDT0001: new disk, initializing [ 2872.849995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2872.868768] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2872.882148] Lustre: 93724:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2872.891081] LustreError: 93724:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2877.296783] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2886.828617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2887.084875] Lustre: lustre-OST0000: new disk, initializing [ 2887.089338] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2887.095312] Lustre: 94489:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2887.175044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2892.972061] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2893.856114] Lustre: 94825:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000a5221411 x1873781826035456/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786977852 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2902.579277] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2909.502945] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2909.509711] Lustre: MGS: Client 9a394c9d-d451-41d9-b873-b9619851748d (at 0@lo) reconnecting [ 2909.558258] Lustre: 92991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2909.575431] LustreError: 93710:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2909.598881] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2909.656906] Lustre: lustre-MDT0000: new disk, initializing [ 2909.769278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2909.794770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2909.814270] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2910.898756] Lustre: 95561:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000006542d5b x1873781826042368/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786977869 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2914.709611] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2916.000761] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2920.060327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2920.075801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2920.133191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2947.046291] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2947.061238] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2947.093592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2947.098731] Lustre: Skipped 1 previous similar message [ 2954.730459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2954.740382] Lustre: Skipped 2 previous similar messages [ 2960.352802] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2960.750746] Lustre: server umount lustre-OST0000 complete [ 2970.083530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2970.087076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2970.093247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2970.108724] Lustre: Skipped 2 previous similar messages [ 2970.125329] Lustre: Skipped 4 previous similar messages [ 2971.946109] Lustre: server umount lustre-MDT0000 complete [ 2975.203067] LustreError: 93696:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2975.222837] LustreError: 93696:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2982.142503] Lustre: server umount lustre-MDT0001 complete [ 2986.143548] Lustre: server umount MGS complete [ 2998.457388] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:45:38 (1786977938) [ 3017.120761] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 3020.212209] Key type lgssc unregistered [ 3020.521157] LNet: 98065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3020.524365] LNetError: 98065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3020.538603] LNet: Removed LNI 192.168.202.137@tcp [ 3021.467483] Key type .llcrypt unregistered [ 3021.469385] Key type ._llcrypt unregistered [ 3040.192819] Key type ._llcrypt registered [ 3040.195131] Key type .llcrypt registered [ 3040.302896] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3041.586119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3041.652213] alg: No test for adler32 (adler32-zlib) [ 3042.948772] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 3043.209129] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 3045.002388] Key type lgssc registered [ 3046.927858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3056.578473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3056.611370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3058.380142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3064.587220] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3067.874976] LustreError: 99289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3067.905610] LustreError: 99289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3073.001072] LustreError: 99288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3074.782227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3075.273579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3079.887981] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3088.357772] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3093.618383] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3100.894228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3101.287857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3103.341573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 3107.491992] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3115.260153] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3126.242298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3126.242855] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3126.258452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3126.273349] Lustre: Skipped 1 previous similar message [ 3131.367926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3131.376944] Lustre: Skipped 1 previous similar message [ 3132.137350] Lustre: server umount lustre-OST0000 complete [ 3136.994361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3136.999605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3137.009356] Lustre: Skipped 1 previous similar message [ 3137.014086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3141.601824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3141.607161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3143.108522] Lustre: server umount lustre-MDT0000 complete [ 3146.721620] LustreError: 100298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3146.735281] LustreError: 100298:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3146.801443] LustreError: 99269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786978088 with bad export cookie 16981582046733350719 [ 3146.805044] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3146.811565] LustreError: 99269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3147.125678] Lustre: server umount lustre-MDT0001 complete [ 3156.853560] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 3159.456983] Key type lgssc unregistered [ 3159.688834] LNet: 102823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3159.694179] LNetError: 102823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3159.711187] LNet: Removed LNI 192.168.202.137@tcp [ 3160.555341] Key type .llcrypt unregistered [ 3160.560559] Key type ._llcrypt unregistered [ 3176.508811] Key type ._llcrypt registered [ 3176.511797] Key type .llcrypt registered [ 3176.652852] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3177.806771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3177.907970] alg: No test for adler32 (adler32-zlib) [ 3178.982275] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 3179.173950] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 3180.839224] Key type lgssc registered [ 3182.019615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3189.367716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3189.372414] LustreError: 103899:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3189.410404] LustreError: 103908:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3189.415872] LustreError: 103899:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4961 [ 3189.421811] LustreError: 103899:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3189.432991] LustreError: 103899:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3189.433261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3190.987519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3200.077910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3200.494376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3206.094598] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3212.955676] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing lsmod [ 3215.723460] LustreError: 105105:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3215.749208] LustreError: 105113:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3215.753194] LustreError: 105105:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4977 [ 3215.762226] LustreError: 105113:0:(obd_mount.c:234:lustre_start_simple()) lustre-OST0000-osd attach error -17 [ 3215.769658] LustreError: 105113:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -17 [ 3215.776885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3215.781256] LustreError: 105113:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 3216.178382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3217.196591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 3220.960805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3220.968378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3221.001674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3222.498542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3222.505409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3225.635907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3225.652810] Lustre: Skipped 4 previous similar messages [ 3230.700399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3230.705629] Lustre: Skipped 2 previous similar messages [ 3233.248040] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3233.451439] Lustre: server umount lustre-MDT0000 complete [ 3235.816028] LustreError: 103945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3235.843954] LustreError: 103945:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3237.369356] LustreError: 103923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786978179 with bad export cookie 5781432449103040914 [ 3237.387910] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3237.943713] LustreError: 104376:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000029d93417 x1873782370229760/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3238.219980] Lustre: server umount lustre-MDT0001 complete [ 3252.364769] Lustre: server umount lustre-OST0000 complete [ 3260.766539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3261.148171] LustreError: 106179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3261.231744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3265.975914] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3266.533303] LustreError: 106180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3271.650369] LustreError: 106179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3274.958448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3275.375404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3280.688586] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3290.226594] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3298.300595] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3306.556849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3306.904416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3314.032796] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3322.033659] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3331.518420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:97) [ 3340.771056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3340.781176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3340.798805] Lustre: Skipped 1 previous similar message [ 3340.807196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3341.819750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3345.356985] Lustre: server umount lustre-OST0000 complete [ 3352.033753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3352.039206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3352.058937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3352.072231] Lustre: Skipped 2 previous similar messages [ 3352.074387] Lustre: Skipped 2 previous similar messages [ 3356.029287] Lustre: server umount lustre-MDT0000 complete [ 3357.152815] LustreError: 106179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3357.161203] LustreError: 106179:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3360.302050] LustreError: 106160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786978302 with bad export cookie 5781432449103041852 [ 3360.309176] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3360.314383] LustreError: 106160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3360.751434] Lustre: server umount lustre-MDT0001 complete [ 3373.462945] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 3376.716389] Key type lgssc unregistered [ 3377.097705] LNet: 109740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3377.113422] LNetError: 109740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3377.133564] LNet: Removed LNI 192.168.202.137@tcp [ 3378.024249] Key type .llcrypt unregistered [ 3378.025903] Key type ._llcrypt unregistered [ 3402.695699] Key type ._llcrypt registered [ 3402.697933] Key type .llcrypt registered [ 3402.827519] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3404.348841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3404.658892] alg: No test for adler32 (adler32-zlib) [ 3405.825577] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 3406.079368] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 3407.775324] Key type lgssc registered [ 3409.257134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3417.665559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3417.686243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3419.269528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3423.360288] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3429.858200] LustreError: 111158:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3429.872808] LustreError: 111158:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3432.608629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3432.961537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3437.652535] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3445.542794] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3451.943549] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3459.202953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3459.593116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3465.793564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3466.694807] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3474.973161] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3483.586267] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3487.420540] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3493.350135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3493.353368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3493.362145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3493.365251] Lustre: Skipped 1 previous similar message [ 3498.478049] LustreError: 113281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3498.535692] Lustre: server umount lustre-OST0000 complete [ 3503.584920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3503.596365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3503.601598] Lustre: Skipped 2 previous similar messages [ 3508.718044] LustreError: 111153:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3508.739332] LustreError: 111153:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3508.784544] Lustre: server umount lustre-MDT0000 complete [ 3512.400498] LustreError: 113000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786978454 with bad export cookie 1267000149460951557 [ 3512.402484] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3512.413525] LustreError: 113000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3512.725504] Lustre: server umount lustre-MDT0001 complete [ 3524.999424] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 3527.999893] Key type lgssc unregistered [ 3528.280403] LNet: 114802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3528.283823] LNetError: 114802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3529.318699] LNet: Removed LNI 192.168.202.137@tcp [ 3530.197122] Key type .llcrypt unregistered [ 3530.199618] Key type ._llcrypt unregistered [ 3549.922025] Key type ._llcrypt registered [ 3549.923346] Key type .llcrypt registered [ 3550.039521] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3551.612561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3551.726150] alg: No test for adler32 (adler32-zlib) [ 3553.029742] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 3553.474597] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 3555.271329] Key type lgssc registered [ 3556.767110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3568.172340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3568.213100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3569.922073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3575.076817] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3579.365061] LustreError: 116023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3579.379323] LustreError: 116023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3584.485661] LustreError: 116024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3586.571514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3586.989923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3592.658256] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3602.742461] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3608.994985] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3617.648819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3618.191660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3626.250932] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3635.120435] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3639.767405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3650.017634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3650.021134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3650.028375] Lustre: Skipped 1 previous similar message [ 3655.137318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3655.149290] Lustre: Skipped 2 previous similar messages [ 3655.687277] Lustre: server umount lustre-OST0000 complete [ 3660.260401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3660.282930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3660.293423] Lustre: Skipped 1 previous similar message [ 3665.379390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3665.382892] Lustre: Skipped 1 previous similar message [ 3666.332862] Lustre: server umount lustre-MDT0000 complete [ 3670.293231] LustreError: 116003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786978612 with bad export cookie 10141603656990196572 [ 3670.312620] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3670.316761] LustreError: 116003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3670.628691] Lustre: server umount lustre-MDT0001 complete [ 3682.422809] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 3685.506918] Key type lgssc unregistered [ 3685.742196] LNet: 119587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3685.750299] LNetError: 119587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3685.764569] LNet: Removed LNI 192.168.202.137@tcp [ 3686.539401] Key type .llcrypt unregistered [ 3686.542216] Key type ._llcrypt unregistered [ 3713.211194] Key type ._llcrypt registered [ 3713.212761] Key type .llcrypt registered [ 3713.360906] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3715.238821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3715.304474] alg: No test for adler32 (adler32-zlib) [ 3716.596318] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 3716.990103] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 3718.679243] Key type lgssc registered [ 3720.147613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3729.039339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3729.059719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3730.797717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3736.018498] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3740.139297] LustreError: 121007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3740.163851] LustreError: 121007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3745.254328] LustreError: 121008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3745.275527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3745.670625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3749.823611] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3758.431958] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3765.838700] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3774.993629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3775.400295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3776.461392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:193) [ 3782.766712] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3792.502512] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3799.944619] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3805.233881] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3805.240846] Lustre: Skipped 2 previous similar messages [ 3806.827599] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3806.881162] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3810.623573] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3810.626334] Lustre: Skipped 2 previous similar messages [ 3815.533497] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3815.536509] Lustre: Skipped 1 previous similar message [ 3821.921621] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.37@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3821.927527] Lustre: Skipped 2 previous similar messages [ 3824.327862] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.37@tcp 192.168.0.[2,10]@tcp [ 3829.025607] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.37/24@tcp in log lustre-MDT0000 [ 3829.031449] Lustre: Skipped 2 previous similar messages [ 3832.752375] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.37/24@tcp [ 3832.757479] Lustre: Skipped 1 previous similar message [ 3839.989677] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3839.998568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3840.031815] LustreError: Skipped 1 previous similar message [ 3840.039234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3842.018923] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3842.033564] LustreError: Skipped 2 previous similar messages [ 3842.037660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3842.039564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3842.089756] Lustre: Skipped 1 previous similar message [ 3846.059580] Lustre: server umount lustre-OST0000 complete [ 3852.256600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3852.277692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3852.286258] Lustre: Skipped 1 previous similar message [ 3856.415197] Lustre: server umount lustre-MDT0000 complete [ 3857.379652] LustreError: 121007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3857.407301] LustreError: 121007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3861.322886] LustreError: 120989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786978803 with bad export cookie 6504539511317184703 [ 3861.327513] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3861.346638] LustreError: 120989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3861.928898] Lustre: server umount lustre-MDT0001 complete [ 3874.440433] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 3877.351530] Key type lgssc unregistered [ 3877.606932] LNet: 125814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3877.618544] LNetError: 125814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3877.636030] LNet: Removed LNI 192.168.202.137@tcp [ 3879.019032] Key type .llcrypt unregistered [ 3879.020754] Key type ._llcrypt unregistered [ 3920.967787] Key type ._llcrypt registered [ 3920.970039] Key type .llcrypt registered [ 3921.040258] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3921.861286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3922.110424] alg: No test for adler32 (adler32-zlib) [ 3923.293603] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 3923.665980] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 3925.400053] Key type lgssc registered [ 3926.582445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3935.076531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3935.097378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3936.689221] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3936.710219] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.37/24@tcp [ 3936.723509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3941.409307] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3945.961464] LustreError: 127390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3945.998883] LustreError: 127390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3950.796188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3951.119095] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3951.146824] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.37/24@tcp [ 3951.203819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3956.552896] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3965.289434] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3972.630857] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3980.861510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3981.436788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3986.935365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:201 to 0x240000400:225) [ 3989.280950] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3998.760616] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4011.508723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4011.540072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4011.542245] Lustre: Skipped 1 previous similar message [ 4011.568878] Lustre: Skipped 1 previous similar message [ 4016.076350] Lustre: server umount lustre-OST0000 complete [ 4021.734947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4021.758919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4026.849574] LustreError: 127390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4027.025911] Lustre: server umount lustre-MDT0000 complete [ 4030.952178] LustreError: 127390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4030.999463] LustreError: 127390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4031.610992] LustreError: 127371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786978973 with bad export cookie 14815059722536957421 [ 4031.616135] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4031.631915] LustreError: 127371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4032.289200] Lustre: server umount lustre-MDT0001 complete [ 4045.739853] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 4049.112500] Key type lgssc unregistered [ 4049.627691] LNet: 130982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4049.662392] LNetError: 130982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4049.693826] LNet: Removed LNI 192.168.202.137@tcp [ 4050.666402] Key type .llcrypt unregistered [ 4050.667893] Key type ._llcrypt unregistered [ 4083.340209] Key type ._llcrypt registered [ 4083.345160] Key type .llcrypt registered [ 4083.487913] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 4100.199919] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4101.682816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4101.709687] alg: No test for adler32 (adler32-zlib) [ 4102.777349] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 4103.056206] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 4104.807165] Key type lgssc registered [ 4106.146971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4152.948910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4152.987588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4154.439790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4154.539542] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4154.720503] Lustre: lustre-MDT0000: new disk, initializing [ 4155.041126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4155.126231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4160.272382] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4174.331619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4174.562792] Lustre: 134387:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4174.694395] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4174.700089] Lustre: Skipped 1 previous similar message [ 4174.828380] Lustre: lustre-MDT0001: new disk, initializing [ 4174.965383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4175.003909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4175.019353] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4180.666952] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4191.826540] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4198.742847] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4206.940941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4207.319952] Lustre: lustre-OST0000: new disk, initializing [ 4207.326951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4207.333408] Lustre: 136355:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4207.441725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4212.833850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4212.844467] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4212.998166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4214.447063] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4225.742287] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4230.821187] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4231.094316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4235.945780] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4236.137891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4238.817209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4238.820847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4238.839880] LustreError: Skipped 1 previous similar message [ 4238.846219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4238.858841] Lustre: Skipped 1 previous similar message [ 4238.886678] Lustre: Skipped 1 previous similar message [ 4243.951559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4249.069180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4249.082541] Lustre: Skipped 2 previous similar messages [ 4252.127187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4252.366644] Lustre: server umount lustre-OST0000 complete [ 4256.737513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4256.744518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4256.771430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4259.298636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4264.416702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4264.432576] Lustre: Skipped 3 previous similar messages [ 4271.071675] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4273.420927] Lustre: server umount lustre-MDT0000 complete [ 4278.003971] LustreError: 135189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786979220 with bad export cookie 8548262322031533251 [ 4278.015382] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4278.019830] LustreError: 135189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4278.604417] Lustre: server umount lustre-MDT0001 complete [ 4289.080380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4289.691070] LustreError: 138579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4289.715058] LustreError: 138579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4289.789738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4295.139412] LustreError: 138580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4296.362409] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4299.234350] LustreError: 138579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4304.354815] LustreError: 138580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4308.470021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4309.007749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4315.135244] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4326.127437] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4333.858695] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4341.792848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4342.366112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4350.478746] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4358.995501] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4364.333329] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4364.737159] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4372.285771] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4372.601437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4384.468210] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4384.757619] Lustre: lustre-OST0001: new disk, initializing [ 4384.762629] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4384.774403] Lustre: 141955:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4384.892261] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4391.711805] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4392.155321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4392.211131] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4392.434371] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4402.300353] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4402.543945] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4408.348138] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4408.573784] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4415.593222] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff931b4990b000.ost_server_uuid 50 [ 4418.339684] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff931b4990b000.ost_server_uuid in IDLE state after 0 sec [ 4433.386111] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4433.409804] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4433.412187] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4433.423073] Lustre: Skipped 1 previous similar message [ 4433.439253] Lustre: Skipped 2 previous similar messages [ 4434.774724] Lustre: server umount lustre-OST0001 complete [ 4438.503638] LustreError: 140405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4438.540368] LustreError: 140405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4442.096938] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4442.111936] LustreError: Skipped 1 previous similar message [ 4442.113914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4445.992878] Lustre: server umount lustre-OST0000 complete [ 4452.323192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4452.333926] LustreError: Skipped 1 previous similar message [ 4452.339381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4452.351870] Lustre: Skipped 1 previous similar message [ 4452.366213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4452.368380] Lustre: Skipped 3 previous similar messages [ 4456.996755] Lustre: server umount lustre-MDT0000 complete [ 4458.987155] LustreError: 139309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4459.013733] LustreError: 139309:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4460.915639] LustreError: 138560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786979403 with bad export cookie 8548262322031534126 [ 4460.923052] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4460.929616] LustreError: 138560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4461.183827] Lustre: server umount lustre-MDT0001 complete [ 4484.687512] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 4487.399275] Key type lgssc unregistered [ 4487.828337] LNet: 144963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4487.839786] LNetError: 144963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4487.859987] LNet: Removed LNI 192.168.202.137@tcp [ 4488.877609] Key type .llcrypt unregistered [ 4488.879451] Key type ._llcrypt unregistered [ 4533.309918] Key type ._llcrypt registered [ 4533.312303] Key type .llcrypt registered [ 4533.422295] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 4548.770718] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4550.166662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4550.206377] alg: No test for adler32 (adler32-zlib) [ 4551.277500] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 4551.537781] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 4553.183181] Key type lgssc registered [ 4554.761204] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4595.141469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4595.171689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4596.514080] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4596.588655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4596.763433] Lustre: lustre-MDT0000: new disk, initializing [ 4596.998080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4597.025606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4602.439772] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4614.395905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4614.533602] Lustre: 149118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4614.652562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4614.659448] Lustre: Skipped 1 previous similar message [ 4614.776238] Lustre: lustre-MDT0001: new disk, initializing [ 4614.888482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4614.916085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4614.928515] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4619.949819] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4632.531955] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4640.456464] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4649.549486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4650.027961] Lustre: lustre-OST0000: new disk, initializing [ 4650.034582] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4650.042439] Lustre: 151087:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4650.175177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4657.245522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4657.254710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4657.375696] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4659.431977] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4670.477438] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4675.511413] Lustre: Failing over lustre-OST0000 [ 4675.800795] Lustre: server umount lustre-OST0000 complete [ 4693.473130] Lustre: 147344:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786979619/real 1786979619] req@0000000003a0f185 x1873783808888320/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1786979635 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4693.526759] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4694.500408] Lustre: 147342:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786979620/real 1786979620] req@00000000d04b9e28 x1873783808888576/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786979636 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4694.569521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4698.015547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4698.365861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4698.380362] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4699.375136] Lustre: 147341:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786979625/real 1786979625] req@000000009dc55dba x1873783808890240/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786979641 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4699.451835] Lustre: 147341:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4703.486367] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4704.544535] Lustre: 147344:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786979630/real 1786979630] req@00000000a48495da x1873783808891520/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786979646 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4704.578627] Lustre: 147344:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4704.640331] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4704.640831] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4708.436745] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4709.408540] Lustre: 147342:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786979635/real 1786979635] req@00000000571f6130 x1873783808892672/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786979651 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4713.588840] Lustre: Failing over lustre-MDT0000 [ 4713.975312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4713.984162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4714.009584] Lustre: Skipped 1 previous similar message [ 4714.034253] Lustre: Skipped 1 previous similar message [ 4714.553147] Lustre: server umount lustre-MDT0000 complete [ 4722.601387] LustreError: 149125:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4722.615337] LustreError: 149125:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4724.196661] LustreError: 149130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4724.219348] LustreError: 149130:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4727.712993] LustreError: 152890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4730.653115] LustreError: 149126:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4730.667829] LustreError: 149126:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4735.466667] Lustre: 147342:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786979661/real 1786979661] req@000000007cfa480c x1873783808906112/t0(0) o400->MGC192.168.202.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786979677 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4735.490963] Lustre: 147342:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4735.496364] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4735.503214] LustreError: 151123:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4735.510542] LustreError: 151123:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4736.479163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4745.123596] LustreError: 152890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4745.149557] LustreError: 152890:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4745.721077] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x842e6bd997b60d0f [ 4745.736385] Lustre: MGC192.168.202.137@tcp: Connection restored to 0@lo (at 0@lo) [ 4745.754249] Lustre: Skipped 1 previous similar message [ 4746.085473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4746.168245] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4748.225200] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4751.363822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4751.415146] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4751.476609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4751.673967] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4761.056142] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4761.060958] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4761.069811] Lustre: Skipped 1 previous similar message [ 4761.075216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4766.707783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4766.714760] Lustre: Skipped 2 previous similar messages [ 4767.271380] Lustre: server umount lustre-OST0000 complete [ 4771.810081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4771.825366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4771.832585] Lustre: Skipped 1 previous similar message [ 4771.834294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4771.860133] Lustre: Skipped 1 previous similar message [ 4776.929112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4776.933764] Lustre: Skipped 1 previous similar message [ 4777.589978] Lustre: server umount lustre-MDT0000 complete [ 4782.054756] LustreError: 150702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4782.068307] LustreError: 150702:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4782.107973] LustreError: 149913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786979724 with bad export cookie 9524668844232543503 [ 4782.112068] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4782.118925] LustreError: 149913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4782.674847] Lustre: server umount lustre-MDT0001 complete [ 4795.606804] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 4799.361588] Key type lgssc unregistered [ 4799.726622] LNet: 154753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4799.733150] LNetError: 154753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4799.752438] LNet: Removed LNI 192.168.202.137@tcp [ 4800.921793] Key type .llcrypt unregistered [ 4800.923262] Key type ._llcrypt unregistered [ 4840.367673] Key type ._llcrypt registered [ 4840.369932] Key type .llcrypt registered [ 4840.460297] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4841.878634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4841.911137] alg: No test for adler32 (adler32-zlib) [ 4843.110638] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 4843.363061] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 4845.119591] Key type lgssc registered [ 4846.666416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4855.496286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4855.509289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4857.172550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4863.109813] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4867.559137] LustreError: 156276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4867.576302] LustreError: 156276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4872.674551] LustreError: 156275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4873.384845] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4877.793367] LustreError: 156276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4877.823064] LustreError: 156304:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4877.834748] LustreError: 156304:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4877.839600] LustreError: 156304:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4878.250879] Lustre: server umount lustre-MDT0000 complete [ 4898.875462] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 4912.571624] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4951.593651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4951.848973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4951.866964] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4951.940279] Lustre: lustre-MDT0000: new disk, initializing [ 4952.016656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4952.030605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4956.595924] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4966.639487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4966.754359] Lustre: 159610:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4966.858978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4966.862928] Lustre: Skipped 1 previous similar message [ 4966.966499] Lustre: lustre-MDT0001: new disk, initializing [ 4967.071831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4967.100727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4967.115507] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4971.727538] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4982.065947] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4987.920669] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4995.001929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4995.380147] Lustre: lustre-OST0000: new disk, initializing [ 4995.385704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4995.397082] Lustre: 161576:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4995.503959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5002.431177] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5002.834579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5002.843829] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5002.946181] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5012.695665] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5016.701497] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5018.890915] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:19:19 (1786979959) [ 5282.795561] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5282.797476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5282.807718] Lustre: Skipped 1 previous similar message [ 5282.832320] Lustre: Skipped 2 previous similar messages [ 5287.907256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5287.914053] Lustre: Skipped 2 previous similar messages [ 5288.210614] Lustre: server umount lustre-MDT0000 complete [ 5291.992166] LustreError: 161587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786980234 with bad export cookie 13715869477821514648 [ 5291.999902] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5292.002782] LustreError: 161587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5293.024516] LustreError: 159617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5293.026455] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5293.074391] Lustre: Skipped 1 previous similar message [ 5293.082062] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5296.040736] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5301.215705] LustreError: 159615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5301.216781] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5301.236246] LustreError: 159615:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5306.335250] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5306.644094] Lustre: server umount lustre-MDT0001 complete [ 5325.279704] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5325.593189] Lustre: server umount lustre-OST0000 complete [ 5335.172741] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 5345.006598] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 5379.553077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5379.783496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5379.808833] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5379.888097] Lustre: lustre-MDT0000: new disk, initializing [ 5379.946485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5379.959213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5384.997474] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5397.522126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5397.638969] Lustre: 166243:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5397.714102] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5397.718712] Lustre: Skipped 1 previous similar message [ 5397.807531] Lustre: lustre-MDT0001: new disk, initializing [ 5397.919971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5397.956663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5397.966849] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5401.743195] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5410.857299] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5418.023266] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5426.294611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5426.685211] Lustre: lustre-OST0000: new disk, initializing [ 5426.693664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5426.707202] Lustre: 168210:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5426.828176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5432.745862] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5434.972665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5434.997146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5435.067076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5443.148990] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5449.196808] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5449.397407] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5455.591601] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5456.007332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5460.972954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5460.981274] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5460.995042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5461.009784] LustreError: Skipped 1 previous similar message [ 5461.036608] Lustre: Skipped 2 previous similar messages [ 5464.890905] Lustre: server umount lustre-OST0000 complete [ 5469.667093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5469.672232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5469.685961] Lustre: Skipped 1 previous similar message [ 5476.028511] Lustre: server umount lustre-MDT0000 complete [ 5476.323906] LustreError: 166255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5476.335986] LustreError: 166255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5480.238151] LustreError: 168207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786980422 with bad export cookie 13715869477821673478 [ 5480.245624] LustreError: 168207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5480.249218] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5481.039398] Lustre: server umount lustre-MDT0001 complete [ 5491.512579] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:27:11 (1786980431) [ 5511.983624] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 5521.471294] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 5564.743358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5564.917581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5564.940242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5564.999574] Lustre: lustre-MDT0000: new disk, initializing [ 5565.094622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5565.112815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5569.404995] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5580.664033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5580.755653] Lustre: 172969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5581.030553] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5585.722873] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5595.310712] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5601.154624] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5607.660480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5607.987466] Lustre: lustre-OST0000: new disk, initializing [ 5607.990493] Lustre: Skipped 1 previous similar message [ 5607.996446] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5608.002569] Lustre: Skipped 2 previous similar messages [ 5608.007642] Lustre: 174949:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5614.811383] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5617.741333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5617.747876] Lustre: Skipped 1 previous similar message [ 5617.750925] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5617.904477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5624.672278] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5633.504444] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5633.512351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5633.514070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5633.523589] Lustre: Skipped 2 previous similar messages [ 5633.527269] Lustre: Skipped 3 previous similar messages [ 5638.882977] Lustre: server umount lustre-OST0000 complete [ 5643.758764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5649.341106] Lustre: server umount lustre-MDT0000 complete [ 5652.661072] LustreError: 172961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786980594 with bad export cookie 13715869477821674360 [ 5652.666949] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5652.682237] LustreError: 172961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5653.019619] Lustre: server umount lustre-MDT0001 complete [ 5662.038751] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:30:02 (1786980602) [ 5682.613720] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 5695.417430] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 5733.819255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5734.102372] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5734.126104] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5734.195708] Lustre: lustre-MDT0000: new disk, initializing [ 5734.270420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5734.273094] Lustre: Skipped 2 previous similar messages [ 5734.287245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5739.295028] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5749.252709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5749.413531] Lustre: 179271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5749.602665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5754.174788] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5762.526886] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5768.220808] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5773.824429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5774.088215] Lustre: 181247:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5780.148569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5780.206371] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5789.964328] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5800.431495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5800.435246] LustreError: Skipped 1 previous similar message [ 5800.436900] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5800.441265] Lustre: Skipped 1 previous similar message [ 5800.443877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5800.446578] Lustre: Skipped 7 previous similar messages [ 5805.426768] Lustre: server umount lustre-OST0000 complete [ 5816.095752] Lustre: server umount lustre-MDT0000 complete [ 5816.288404] LustreError: 179283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5816.317947] LustreError: 179283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5820.570069] LustreError: 179264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786980762 with bad export cookie 13715869477821675445 [ 5820.576400] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5820.581532] LustreError: 179264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5820.910581] Lustre: server umount lustre-MDT0001 complete [ 5832.610882] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 5835.411533] Key type lgssc unregistered [ 5835.682740] LNet: 183106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5835.686952] LNetError: 183106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5835.698209] LNet: Removed LNI 192.168.202.137@tcp [ 5836.471640] Key type .llcrypt unregistered [ 5836.474590] Key type ._llcrypt unregistered [ 5862.340473] Key type ._llcrypt registered [ 5862.342554] Key type .llcrypt registered [ 5862.434035] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 5863.578087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5863.693774] alg: No test for adler32 (adler32-zlib) [ 5864.763691] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 5865.161187] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 5866.919419] Key type lgssc registered [ 5868.118678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5876.569772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5876.612601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5878.282365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5883.666059] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5888.487352] LustreError: 184523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5888.518609] LustreError: 184523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5893.610779] LustreError: 184524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5893.813911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5894.385957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5899.058919] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5907.579194] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5914.777194] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5921.609066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5922.109943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5924.164089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5928.319547] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5935.080257] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5960.165500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5960.167278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5960.176265] LustreError: Skipped 1 previous similar message [ 5960.180652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5960.192019] Lustre: Skipped 1 previous similar message [ 5960.209651] Lustre: Skipped 1 previous similar message [ 5963.451515] Lustre: server umount lustre-OST0000 complete [ 5970.400431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5970.407986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5970.416016] Lustre: Skipped 1 previous similar message [ 5970.432351] Lustre: Skipped 1 previous similar message [ 5973.948112] Lustre: server umount lustre-MDT0000 complete [ 5975.521512] LustreError: 185537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5975.544372] LustreError: 185537:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5977.397625] LustreError: 184503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786980919 with bad export cookie 8545008683066718831 [ 5977.401956] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5977.407455] LustreError: 184503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5977.746608] Lustre: server umount lustre-MDT0001 complete [ 5989.523523] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 5991.920574] Key type lgssc unregistered [ 5992.156396] LNet: 188087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5992.162553] LNetError: 188087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5993.189688] LNet: Removed LNI 192.168.202.137@tcp [ 5993.849328] Key type .llcrypt unregistered [ 5993.851215] Key type ._llcrypt unregistered [ 6017.115215] Key type ._llcrypt registered [ 6017.117439] Key type .llcrypt registered [ 6017.193574] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 6018.112848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6018.203792] alg: No test for adler32 (adler32-zlib) [ 6019.180774] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 6019.365221] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 6021.009220] Key type lgssc registered [ 6022.226749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6029.029962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6029.041485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6030.596154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6035.006354] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6040.040855] LustreError: 189505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6040.063512] LustreError: 189505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6043.329954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6043.590764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6048.004895] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6054.934378] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6060.587359] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6067.826228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6068.231505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6073.930101] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6075.384385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6080.951246] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6088.150771] Lustre: lustre-OST0000: Not available for connect from 192.168.202.37@tcp (stopping) [ 6089.700235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6089.722099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6089.737291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6090.721050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6090.734957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6090.745058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6090.747700] Lustre: Skipped 1 previous similar message [ 6093.148713] Lustre: server umount lustre-OST0000 complete [ 6100.955866] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff931b45cf2000.ost_server_uuid 50 [ 6102.584779] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6120.419118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6120.419736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6120.430743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6120.437746] Lustre: Skipped 1 previous similar message [ 6120.455282] Lustre: Skipped 1 previous similar message [ 6122.782765] Lustre: server umount lustre-MDT0000 complete [ 6125.536812] LustreError: 189501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6125.569453] LustreError: 189501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6126.494031] LustreError: 189486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786981068 with bad export cookie 16259577423085313569 [ 6126.500825] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6126.886676] Lustre: server umount lustre-MDT0001 complete [ 6137.077798] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:37:57 (1786981077) [ 6142.744113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6143.157589] LustreError: 193428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6143.215674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6147.766601] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6148.581756] LustreError: 193429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6153.697508] LustreError: 193428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6156.000240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6156.377913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6160.289858] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6168.505756] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6174.513170] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6181.513600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6181.803027] LustreError: 195249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6182.058423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6188.418574] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6188.538872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6197.331920] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6205.303647] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6205.553796] Lustre: lustre-OST0001: new disk, initializing [ 6205.558144] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6205.562485] Lustre: 196271:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6205.685832] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6212.725130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6212.743956] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6212.899251] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6213.647555] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6224.339875] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6228.453116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6228.477466] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6228.516647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6228.530523] Lustre: Skipped 1 previous similar message [ 6233.498422] Lustre: server umount lustre-OST0000 complete [ 6233.571206] LustreError: 196279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6233.591641] LustreError: 196279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6239.251969] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6239.447852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6244.362961] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6244.572208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6259.172811] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6259.175938] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6259.187187] LustreError: Skipped 1 previous similar message [ 6259.187947] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6259.187951] Lustre: Skipped 1 previous similar message [ 6259.226028] Lustre: Skipped 2 previous similar messages [ 6264.101766] Lustre: server umount lustre-OST0001 complete [ 6268.899597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6268.904674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6279.652053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6279.663263] Lustre: Skipped 7 previous similar messages [ 6282.207409] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6282.347226] Lustre: server umount lustre-MDT0000 complete [ 6284.772166] LustreError: 193429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6284.803090] LustreError: 193429:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 6285.426142] LustreError: 196269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786981227 with bad export cookie 16259577423085314640 [ 6285.433337] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6285.439037] LustreError: 196269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6285.768910] Lustre: server umount lustre-MDT0001 complete [ 6309.760613] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:40:50 (1786981250) [ 6317.489548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6317.852222] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6317.889355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6318.108725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6322.643942] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6332.809381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6332.955769] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6333.043489] Lustre: 199711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6339.269975] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6348.632279] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6355.155712] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6362.543966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6362.680229] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6369.621164] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6371.050234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6379.896473] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6388.179756] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6388.265585] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6388.484582] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6388.495338] Lustre: Skipped 2 previous similar messages [ 6395.018632] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6395.416327] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6403.003415] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6407.136270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6407.139101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6407.145943] LustreError: Skipped 1 previous similar message [ 6407.163349] Lustre: Skipped 1 previous similar message [ 6407.166974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6412.365634] Lustre: server umount lustre-OST0000 complete [ 6414.251363] LustreError: 201564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6433.898701] Lustre: server umount lustre-OST0001 complete [ 6440.932972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6440.947285] LustreError: Skipped 2 previous similar messages [ 6440.950635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6440.966477] Lustre: Skipped 3 previous similar messages [ 6452.191414] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6452.683335] Lustre: server umount lustre-MDT0000 complete [ 6457.216549] LustreError: 199705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786981399 with bad export cookie 16259577423085316040 [ 6457.230067] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6457.598795] Lustre: server umount lustre-MDT0001 complete [ 6483.278899] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:43:43 (1786981423) [ 6501.656475] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 6518.734842] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 6560.167571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6560.419099] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6560.476084] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6560.558146] Lustre: lustre-MDT0000: new disk, initializing [ 6560.653365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6560.680207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6564.945708] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6575.736771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6575.825565] Lustre: 207832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6575.875942] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6575.879981] Lustre: Skipped 1 previous similar message [ 6575.944134] Lustre: lustre-MDT0001: new disk, initializing [ 6576.082274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6576.094132] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6580.414071] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6590.203855] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6597.047711] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6605.156558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6605.400476] Lustre: lustre-OST0000: new disk, initializing [ 6605.404234] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6605.409849] Lustre: 209799:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6614.899678] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6615.145507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6615.152664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6615.317917] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6626.446757] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6633.210542] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6633.695171] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6639.264681] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6639.650232] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6645.228869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6645.279870] Lustre: Skipped 1 previous similar message [ 6645.314756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6645.327615] Lustre: Skipped 15 previous similar messages [ 6648.539426] Lustre: server umount lustre-OST0000 complete [ 6652.919598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6659.477495] Lustre: server umount lustre-MDT0000 complete [ 6660.582702] LustreError: 207844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6660.614481] LustreError: 207844:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6663.559050] LustreError: 207826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786981605 with bad export cookie 16259577423085317832 [ 6663.567182] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6663.579110] LustreError: 207826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6664.135306] Lustre: server umount lustre-MDT0001 complete [ 6673.540038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6679.461510] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6688.868111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6694.492534] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6703.109170] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6709.235759] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6717.343385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6724.997829] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6734.711679] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6739.908770] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6740.132815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6745.320467] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6745.585460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6753.965983] Lustre: server umount lustre-OST0000 complete [ 6762.011717] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6767.568335] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6797.211517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6804.638737] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6813.164698] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6815.167187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6815.208942] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6815.214492] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6815.267669] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6820.324600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6820.330346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6820.333026] LustreError: Skipped 1 previous similar message [ 6820.348113] Lustre: Skipped 5 previous similar messages [ 6820.350939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6820.365172] Lustre: Skipped 6 previous similar messages [ 6825.821971] Lustre: server umount lustre-OST0000 complete [ 6836.495642] Lustre: server umount lustre-MDT0000 complete [ 6840.994972] LustreError: 212002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786981783 with bad export cookie 16259577423085318714 [ 6841.008114] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6841.441106] Lustre: server umount lustre-MDT0001 complete [ 6851.775461] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:49:51 (1786981791) [ 6860.573785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6861.296465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6861.310578] Lustre: Skipped 6 previous similar messages [ 6866.468437] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6875.821385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6881.302605] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6889.953416] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6896.919915] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6906.604642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6913.813205] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6922.119116] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6927.150396] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6931.855941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 6936.475848] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6936.711871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6943.555471] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6943.727394] Lustre: lustre-OST0001: new disk, initializing [ 6943.730849] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6943.741947] Lustre: 221390:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6952.058442] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6952.122517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6952.145691] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6952.369312] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6962.493925] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6967.545248] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6967.748288] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6972.033681] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6972.200372] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6988.255213] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6988.369294] Lustre: server umount lustre-OST0001 complete [ 6993.396168] LustreError: 219811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6993.409482] LustreError: 219811:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6996.194314] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6996.550034] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7002.417979] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7002.652315] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7032.441285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7037.933390] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7037.950714] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7037.953149] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7037.981266] Lustre: Skipped 2 previous similar messages [ 7039.672815] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7048.208417] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7050.606725] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7059.632097] Lustre: server umount lustre-OST0001 complete [ 7078.881816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7078.905917] Lustre: Skipped 7 previous similar messages [ 7078.926603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7078.932313] Lustre: Skipped 16 previous similar messages [ 7084.630505] Lustre: server umount lustre-OST0000 complete [ 7091.168551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7091.175525] LustreError: Skipped 4 previous similar messages [ 7095.517916] Lustre: server umount lustre-MDT0000 complete [ 7099.821664] LustreError: 217970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786982041 with bad export cookie 16259577423085319680 [ 7099.830161] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7099.842487] LustreError: 217970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7114.208964] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7114.770537] Lustre: server umount lustre-MDT0001 complete [ 7145.519422] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:54:46 (1786982086) [ 7152.519951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7152.795478] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7152.828718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7157.431438] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7167.741514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7167.906528] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7167.933856] Lustre: 226637:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7173.097832] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7183.326561] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7190.159357] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7198.713657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7198.885709] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7205.592569] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7213.332463] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7223.192778] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7223.313044] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7230.868307] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7243.160912] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7246.327977] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7252.972766] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7253.154215] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7258.226688] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7258.491354] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7265.680861] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff931b44d87000.ost_server_uuid 50 [ 7267.166800] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff931b44d87000.ost_server_uuid in IDLE state after 0 sec [ 7268.364454] Lustre: Permanently deactivating lustre-OST0001 [ 7268.378861] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7271.092474] Lustre: Permanently reactivating lustre-OST0001 [ 7280.916607] Lustre: server umount lustre-OST0001 complete [ 7292.108975] Lustre: server umount lustre-OST0000 complete [ 7303.245474] Lustre: server umount lustre-MDT0000 complete [ 7307.504855] LustreError: 226631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786982249 with bad export cookie 16259577423085320968 [ 7307.520129] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7307.852809] Lustre: server umount lustre-MDT0001 complete [ 7331.510976] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:57:51 (1786982271) [ 7340.171223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7340.625540] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7340.668249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7340.679797] Lustre: Skipped 5 previous similar messages [ 7346.159183] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7355.534159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7355.633104] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7355.686869] Lustre: 233334:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7360.274442] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7368.154191] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7373.599845] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7380.092183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7380.261750] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7380.281797] Lustre: Permanently deactivating lustre-OST0000 [ 7380.291891] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7380.468536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7380.490664] Lustre: Skipped 10 previous similar messages [ 7386.405605] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7389.764742] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7393.472377] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7399.042885] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7399.090137] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7405.214246] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7407.271266] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7413.292292] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7415.050772] Lustre: Permanently reactivating lustre-OST0000 [ 7415.063902] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7415.072450] Lustre: Skipped 2 previous similar messages [ 7416.205098] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7416.215699] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7416.227409] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7416.305743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7416.325129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7428.620570] Lustre: server umount lustre-OST0001 complete [ 7434.060385] Lustre: server umount lustre-OST0000 complete [ 7444.291716] LustreError: 237436:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000000835af2f) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7446.452518] Lustre: server umount lustre-MDT0000 complete [ 7450.237264] LustreError: 233324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786982392 with bad export cookie 16259577423085322613 [ 7450.251341] LustreError: 233324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7450.255851] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7450.601128] Lustre: server umount lustre-MDT0001 complete [ 7463.834928] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 7466.441059] Key type lgssc unregistered [ 7466.697458] LNet: 238191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7466.718674] LNetError: 238191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7466.733935] LNet: Removed LNI 192.168.202.137@tcp [ 7467.431218] Key type .llcrypt unregistered [ 7467.433532] Key type ._llcrypt unregistered [ 7491.794638] Key type ._llcrypt registered [ 7491.797695] Key type .llcrypt registered [ 7491.892535] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 7493.202088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7493.219369] alg: No test for adler32 (adler32-zlib) [ 7494.207395] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 7494.387214] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 7496.031707] Key type lgssc registered [ 7496.904609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7506.448690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7506.460739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7507.977570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7513.075699] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7518.193608] LustreError: 239659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7518.235703] LustreError: 239659:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7523.301381] LustreError: 239658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7525.623728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7525.821722] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7525.871560] Lustre: 239647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7525.889406] Lustre: Permanently deactivating lustre-MDT0001 [ 7525.924711] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7526.499206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7532.590873] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7533.136831] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7542.436768] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7549.006608] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7555.353461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7555.669746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7564.358626] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7571.044237] LustreError: 241513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7571.062976] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7573.061701] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7576.555058] LustreError: 241513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7578.709769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7583.749585] LustreError: 241514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7583.790127] LustreError: 241514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7584.969462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7585.363397] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7590.411116] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7590.417345] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7594.682472] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7604.084850] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7608.138658] Lustre: Permanently reactivating lustre-MDT0001 [ 7608.157535] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7608.161587] Lustre: Skipped 2 previous similar messages [ 7609.636450] Lustre: lustre-MDT0001: Client e0c1793c-491e-4661-b887-8c9f930be600 (at 192.168.202.37@tcp) reconnecting [ 7616.666156] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7616.688683] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7616.698957] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7616.712040] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7618.706908] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7620.439865] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7625.496479] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7625.782852] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7630.028549] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7630.326311] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7632.881943] Lustre: Permanently deactivating lustre-MDT0001 [ 7632.905748] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7632.911821] Lustre: Skipped 2 previous similar messages [ 7642.252448] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7654.375590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7654.383656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7654.415548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7656.930082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7656.934040] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7656.938904] Lustre: Skipped 1 previous similar message [ 7656.949056] Lustre: Skipped 2 previous similar messages [ 7658.325780] Lustre: server umount lustre-MDT0000 complete [ 7662.051368] LustreError: 239659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7662.063960] LustreError: 239659:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7662.753838] LustreError: 240409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786982604 with bad export cookie 18345098530387365044 [ 7662.755038] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7662.781419] LustreError: 240409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7663.557672] Lustre: server umount lustre-MDT0001 complete [ 7670.185144] Lustre: server umount lustre-OST0000 complete [ 7682.305070] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:03:42 (1786982622) [ 7693.846589] Lustre: server umount lustre-OST0001 complete [ 7701.030175] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 7710.009487] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 7745.603860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7745.892301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7745.904977] Lustre: Skipped 2 previous similar messages [ 7745.940895] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7746.039776] Lustre: lustre-MDT0000: new disk, initializing [ 7746.095307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7746.112508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7751.107423] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7761.068497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7761.166421] Lustre: 247977:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7761.211363] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7761.215816] Lustre: Skipped 1 previous similar message [ 7761.287577] Lustre: lustre-MDT0001: new disk, initializing [ 7761.396532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7761.441418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7761.452883] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7765.670709] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7774.336163] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7780.440829] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7788.247431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7788.518674] Lustre: lustre-OST0000: new disk, initializing [ 7788.522176] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7788.531505] Lustre: 249950:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7788.616391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7794.254194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7794.269081] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7794.412110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7794.763749] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7804.321782] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7806.786687] LustreError: 247987:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7816.801324] LustreError: 247987:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7816.817851] LustreError: 250668:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7817.211705] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7817.490358] Lustre: lustre-OST0001: new disk, initializing [ 7817.500458] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7817.511787] Lustre: 251064:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7824.980405] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7825.512317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7825.524738] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7825.600210] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7826.905055] LustreError: 250668:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7836.280431] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7838.031615] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7841.251622] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7841.280052] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7841.303662] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7841.326075] Lustre: Skipped 1 previous similar message [ 7841.359888] Lustre: Skipped 1 previous similar message [ 7846.369268] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7846.371891] Lustre: Skipped 1 previous similar message [ 7851.513248] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7851.528243] Lustre: Skipped 2 previous similar messages [ 7855.074081] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7855.185637] Lustre: server umount lustre-OST0001 complete [ 7856.610120] LustreError: 250271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7856.630675] LustreError: 250271:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7876.579216] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7876.581455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7876.590911] LustreError: Skipped 2 previous similar messages [ 7876.592457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7876.613484] Lustre: Skipped 1 previous similar message [ 7876.630151] Lustre: Skipped 2 previous similar messages [ 7886.303332] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7886.586822] Lustre: server umount lustre-OST0000 complete [ 7892.449897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7897.570749] LustreError: 247986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7897.595143] LustreError: 247986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 7898.750380] Lustre: server umount lustre-MDT0000 complete [ 7902.955390] LustreError: 247971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786982845 with bad export cookie 18345098530387367858 [ 7902.963940] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7902.967584] LustreError: 247971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7909.445260] Lustre: server umount lustre-MDT0001 complete [ 7921.724633] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 7924.595956] Key type lgssc unregistered [ 7924.918731] LNet: 253102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7924.922987] LNetError: 253102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7924.944431] LNet: Removed LNI 192.168.202.137@tcp [ 7925.728080] Key type .llcrypt unregistered [ 7925.729839] Key type ._llcrypt unregistered [ 7969.524611] Key type ._llcrypt registered [ 7969.528956] Key type .llcrypt registered [ 7969.695626] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 7971.227912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7971.258115] alg: No test for adler32 (adler32-zlib) [ 7972.494339] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 7972.847915] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 7974.631377] Key type lgssc registered [ 7975.814669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7983.383814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7983.442124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7984.824566] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7984.869494] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7985.090880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7989.913643] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7999.301365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7999.516269] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7999.575896] Lustre: 255311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7999.951682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8005.631229] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8014.680897] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8021.670871] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8030.205163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8030.515044] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8030.776750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8037.641488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8038.308750] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8047.284365] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8104.426433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8104.441846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8104.447911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8104.456914] Lustre: Skipped 1 previous similar message [ 8104.474532] Lustre: Skipped 1 previous similar message [ 8108.952641] Lustre: server umount lustre-OST0000 complete [ 8113.532486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8125.637856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8126.078986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8132.201185] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8140.178372] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8144.433846] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8144.460525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8144.465456] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8144.497712] Lustre: Skipped 1 previous similar message [ 8180.204636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8180.204839] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8180.224659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8180.227639] LustreError: Skipped 1 previous similar message [ 8180.236851] Lustre: Skipped 1 previous similar message [ 8185.331119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8185.339834] Lustre: Skipped 1 previous similar message [ 8185.916045] Lustre: server umount lustre-OST0000 complete [ 8190.444080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8190.469238] Lustre: Skipped 1 previous similar message [ 8190.479533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8190.493271] Lustre: Skipped 1 previous similar message [ 8195.568389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8195.576518] Lustre: Skipped 1 previous similar message [ 8204.255201] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8204.718043] Lustre: server umount lustre-MDT0000 complete [ 8209.517422] LustreError: 255305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786983151 with bad export cookie 1860474423986915492 [ 8209.532784] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8209.541337] LustreError: 255305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8210.141776] Lustre: server umount lustre-MDT0001 complete [ 8223.647399] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 8226.881538] Key type lgssc unregistered [ 8227.253429] LNet: 260741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8227.261458] LNetError: 260741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8228.329355] LNet: Removed LNI 192.168.202.137@tcp [ 8229.092660] Key type .llcrypt unregistered [ 8229.094508] Key type ._llcrypt unregistered [ 8260.372677] Key type ._llcrypt registered [ 8260.379272] Key type .llcrypt registered [ 8260.526569] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 8262.031687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8262.198323] alg: No test for adler32 (adler32-zlib) [ 8263.434339] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 8263.945125] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 8265.759317] Key type lgssc registered [ 8267.422897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8276.967615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8276.983208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8278.372455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8283.656856] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8288.746300] LustreError: 262162:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8288.781385] LustreError: 262162:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8293.684817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8293.859514] LustreError: 262161:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8294.152696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8298.922992] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8308.593281] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8316.301634] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8326.142808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8326.732587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8335.316785] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8344.616864] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8348.277065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8375.265856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8375.272678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8375.286112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8379.359992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8379.365777] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8379.384938] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8381.139045] Lustre: server umount lustre-OST0000 complete [ 8386.530336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8386.537491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8386.558635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8386.561057] Lustre: Skipped 1 previous similar message [ 8389.602184] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8389.602333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8389.639215] Lustre: Skipped 1 previous similar message [ 8390.803409] Lustre: server umount lustre-MDT0000 complete [ 8394.315377] LustreError: 262141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786983336 with bad export cookie 18431016820850704299 [ 8394.326147] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8394.877198] Lustre: server umount lustre-MDT0001 complete [ 8406.988492] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 8409.512412] Key type lgssc unregistered [ 8409.802126] LNet: 266473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8409.808294] LNetError: 266473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8409.830645] LNet: Removed LNI 192.168.202.137@tcp [ 8410.732528] Key type .llcrypt unregistered [ 8410.737313] Key type ._llcrypt unregistered [ 8427.745594] Key type ._llcrypt registered [ 8427.746868] Key type .llcrypt registered [ 8427.844482] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 8429.064626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8429.224410] alg: No test for adler32 (adler32-zlib) [ 8430.232830] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 8430.430927] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 8432.119180] Key type lgssc registered [ 8433.116824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8441.473619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8441.515534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8442.839465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8446.992988] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8452.139109] LustreError: 267694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8452.159737] LustreError: 267694:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8456.433796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8456.851362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8460.483927] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8467.641821] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8474.059785] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8480.586615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8480.880431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8487.094434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8487.887230] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8495.923498] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8507.872733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8507.875677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8507.897040] LustreError: Skipped 1 previous similar message [ 8507.902450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8507.911571] Lustre: Skipped 1 previous similar message [ 8507.931697] Lustre: Skipped 1 previous similar message [ 8512.996977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8513.013982] Lustre: Skipped 1 previous similar message [ 8513.272819] Lustre: server umount lustre-OST0000 complete [ 8518.115677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8518.123539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8518.125118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8518.145959] Lustre: Skipped 1 previous similar message [ 8523.234639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8523.241328] Lustre: Skipped 1 previous similar message [ 8523.738660] Lustre: server umount lustre-MDT0000 complete [ 8527.720931] LustreError: 268429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786983469 with bad export cookie 8758648745275640075 [ 8527.728735] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8527.733196] LustreError: 268429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8528.072177] Lustre: server umount lustre-MDT0001 complete [ 8540.869097] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 8543.559830] Key type lgssc unregistered [ 8543.931625] LNet: 271375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8543.938815] LNetError: 271375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8543.953864] LNet: Removed LNI 192.168.202.137@tcp [ 8544.871371] Key type .llcrypt unregistered [ 8544.874209] Key type ._llcrypt unregistered [ 8571.044900] Key type ._llcrypt registered [ 8571.047182] Key type .llcrypt registered [ 8571.169822] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 8572.295612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8572.345699] alg: No test for adler32 (adler32-zlib) [ 8573.570381] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 8573.922164] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 8575.631174] Key type lgssc registered [ 8576.949963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8585.504487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8585.526549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8587.054933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8591.615716] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8597.478712] LustreError: 272793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8597.491586] LustreError: 272793:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8600.255686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8600.560845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8605.407364] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8613.376560] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8620.426859] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8628.111066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8628.471772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8633.591937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8635.029946] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8642.344653] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8669.670418] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8669.676701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8669.687025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8671.718664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8671.726281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8675.527926] Lustre: server umount lustre-OST0000 complete [ 8681.953946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8681.975904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8681.991068] Lustre: Skipped 2 previous similar messages [ 8685.700937] Lustre: server umount lustre-MDT0000 complete [ 8687.088284] LustreError: 272788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8687.111779] LustreError: 272788:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8690.156354] LustreError: 274636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786983632 with bad export cookie 6991434255184541008 [ 8690.166599] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8690.176167] LustreError: 274636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8690.690226] Lustre: server umount lustre-MDT0001 complete [ 8705.764787] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 8709.643319] Key type lgssc unregistered [ 8709.892730] LNet: 277104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8709.914803] LNetError: 277104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8709.925757] LNet: Removed LNI 192.168.202.137@tcp [ 8711.107746] Key type .llcrypt unregistered [ 8711.111594] Key type ._llcrypt unregistered [ 8729.111359] Key type ._llcrypt registered [ 8729.118417] Key type .llcrypt registered [ 8729.258235] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 8730.468258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8730.496643] alg: No test for adler32 (adler32-zlib) [ 8731.558458] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 8731.778821] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 8733.447311] Key type lgssc registered [ 8734.544614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8741.862118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8741.888696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8743.443350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8747.796643] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8753.635190] LustreError: 278340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8753.658880] LustreError: 278340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8756.364346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8756.778885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8761.702419] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8769.535501] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8775.809742] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8782.731472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8783.107861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8788.335387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8788.775880] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8796.680179] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8812.001761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8812.007184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8812.025068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8812.030822] Lustre: Skipped 1 previous similar message [ 8812.042456] Lustre: Skipped 1 previous similar message [ 8817.121086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8817.134155] Lustre: Skipped 1 previous similar message [ 8818.461832] Lustre: server umount lustre-OST0000 complete [ 8827.364552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8827.393909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8827.395648] Lustre: Skipped 1 previous similar message [ 8827.427167] Lustre: Skipped 1 previous similar message [ 8829.911020] Lustre: server umount lustre-MDT0000 complete [ 8832.500778] LustreError: 278348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8832.518327] LustreError: 278348:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8834.113364] LustreError: 278305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786983776 with bad export cookie 7172340915019369691 [ 8834.135508] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8834.767177] Lustre: server umount lustre-MDT0001 complete [ 8846.695061] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 8849.443049] Key type lgssc unregistered [ 8849.692814] LNet: 282045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8849.698443] LNetError: 282045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8849.725638] LNet: Removed LNI 192.168.202.137@tcp [ 8850.561453] Key type .llcrypt unregistered [ 8850.570856] Key type ._llcrypt unregistered [ 8863.290077] Key type ._llcrypt registered [ 8863.296581] Key type .llcrypt registered [ 8863.485847] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8883.560285] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 8899.075981] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 8900.035904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8900.138143] alg: No test for adler32 (adler32-zlib) [ 8901.112598] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 8901.297271] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 8902.943438] Key type lgssc registered [ 8903.901720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8947.697798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8947.719360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8949.198982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8949.288140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8949.454299] Lustre: lustre-MDT0000: new disk, initializing [ 8949.600710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8949.623750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8955.282868] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8965.583119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8965.744434] Lustre: 285602:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8965.800986] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8965.804470] Lustre: Skipped 1 previous similar message [ 8965.859624] Lustre: lustre-MDT0001: new disk, initializing [ 8965.945524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8965.981578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8965.993670] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8970.735267] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8979.138692] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8985.066253] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8993.050543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8993.310763] Lustre: lustre-OST0000: new disk, initializing [ 8993.315348] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8993.328249] Lustre: 287571:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8993.459638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9000.562433] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9002.592287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9002.614259] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9002.737561] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9010.429679] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9015.023628] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9015.212657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9019.285819] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9019.521966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9023.462615] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9023.465138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9023.484648] LustreError: Skipped 1 previous similar message [ 9023.485946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9023.542168] Lustre: Skipped 1 previous similar message [ 9027.680473] Lustre: server umount lustre-OST0000 complete [ 9033.698977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9033.719001] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9033.721012] Lustre: Skipped 1 previous similar message [ 9033.740353] Lustre: Skipped 2 previous similar messages [ 9038.818220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9038.833474] Lustre: Skipped 1 previous similar message [ 9039.432839] Lustre: server umount lustre-MDT0000 complete [ 9043.141649] LustreError: 285595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786983985 with bad export cookie 16758489210630160557 [ 9043.149060] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9043.150064] LustreError: 285595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9043.543594] Lustre: server umount lustre-MDT0001 complete [ 9054.320668] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 12:26:34 (1786983994) [ 9061.474924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9061.946040] LustreError: 289983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9061.971921] LustreError: 289983:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9062.032417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9066.109391] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9067.488389] LustreError: 289984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9067.509398] LustreError: 289984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9072.610488] LustreError: 289983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9076.333723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9076.775640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9082.131681] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9090.590441] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9097.521950] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9105.323562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9105.756927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9113.444850] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9122.553625] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9136.611830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9136.612729] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9136.641425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9141.732982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9141.740818] Lustre: Skipped 1 previous similar message [ 9142.315666] Lustre: server umount lustre-OST0000 complete [ 9148.385828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9148.403087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9148.436147] Lustre: Skipped 1 previous similar message [ 9151.971205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9151.988420] Lustre: Skipped 3 previous similar messages [ 9162.208269] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9162.796565] Lustre: server umount lustre-MDT0000 complete [ 9166.314746] LustreError: 289978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9166.358936] LustreError: 289978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9168.347806] LustreError: 291828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786984110 with bad export cookie 16758489210630161439 [ 9168.361400] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9168.368606] LustreError: 291828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9168.844927] Lustre: server umount lustre-MDT0001 complete [ 9185.203809] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 9188.063805] Key type lgssc unregistered [ 9188.276634] LNet: 293548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9188.281361] LNetError: 293548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9188.290870] LNet: Removed LNI 192.168.202.137@tcp [ 9189.098192] Key type .llcrypt unregistered [ 9189.100686] Key type ._llcrypt unregistered [ 9219.421342] Key type ._llcrypt registered [ 9219.423644] Key type .llcrypt registered [ 9219.537572] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 9237.679028] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 9239.054969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9239.145309] alg: No test for adler32 (adler32-zlib) [ 9240.496936] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 9240.917291] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 9242.607255] Key type lgssc registered [ 9244.036152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9302.885293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9302.910809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9304.378175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9304.429097] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9304.570933] Lustre: lustre-MDT0000: new disk, initializing [ 9304.744196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9304.761659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9308.635080] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9321.137649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9321.315990] Lustre: 297376:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9321.361653] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9321.366529] Lustre: Skipped 1 previous similar message [ 9321.431523] Lustre: lustre-MDT0001: new disk, initializing [ 9321.512702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9321.538420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9321.553559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9326.812584] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9338.237598] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9345.807964] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9353.683375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9354.061115] Lustre: lustre-OST2710: new disk, initializing [ 9354.065966] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9354.093187] Lustre: 299348:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9354.207513] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9360.921543] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9361.996729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9362.013336] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9362.186253] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9372.436117] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9379.565322] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9379.838223] Lustre: lustre-OST03e8: new disk, initializing [ 9379.841849] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9379.847124] Lustre: 300412:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9379.945481] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9387.482208] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9389.705682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9389.720640] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9389.857205] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9397.762355] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9408.081776] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9408.346667] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9413.367645] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9413.559895] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9419.502330] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9419.904961] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9426.054505] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9426.236955] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9432.303824] Lustre: server umount lustre-MDT0000 complete [ 9434.080791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9434.087442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9436.142125] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9436.614839] LustreError: 297370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786984378 with bad export cookie 8799353089258749850 [ 9436.647756] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9437.256824] Lustre: server umount lustre-MDT0001 complete [ 9452.470130] Lustre: server umount lustre-OST2710 complete [ 9468.001082] Lustre: server umount lustre-OST03e8 complete [ 9493.298872] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 9506.180393] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 9550.917699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9551.247690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9551.270345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9551.347656] Lustre: lustre-MDT0000: new disk, initializing [ 9551.435062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9551.452216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9557.620836] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9569.702516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9569.861033] Lustre: 305669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9569.927247] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9569.931460] Lustre: Skipped 1 previous similar message [ 9570.064081] Lustre: lustre-MDT0001: new disk, initializing [ 9570.146245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9570.183349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9570.193711] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9574.845586] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9586.543569] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9593.903988] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9603.896974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9604.174445] Lustre: lustre-OST0000: new disk, initializing [ 9604.178223] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9604.184360] Lustre: 307635:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9604.315655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9611.951584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9611.996746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9612.240128] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9615.107880] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9626.500729] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9632.993924] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9633.234804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9639.750995] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9639.982728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9642.984916] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9642.988779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9642.988793] Lustre: Skipped 2 previous similar messages [ 9642.990034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9643.027413] LustreError: Skipped 1 previous similar message [ 9648.097953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9648.115120] Lustre: Skipped 2 previous similar messages [ 9649.127281] Lustre: server umount lustre-OST0000 complete [ 9653.216184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9653.224940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9653.248147] Lustre: Skipped 2 previous similar messages [ 9653.272122] Lustre: Skipped 1 previous similar message [ 9658.347681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9658.357523] Lustre: Skipped 1 previous similar message [ 9659.599093] Lustre: server umount lustre-MDT0000 complete [ 9663.464810] LustreError: 305677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9663.477608] LustreError: 305677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 9664.067120] LustreError: 305663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786984606 with bad export cookie 8799353089258751530 [ 9664.079413] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9664.089619] LustreError: 305663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9664.651413] Lustre: server umount lustre-MDT0001 complete [ 9675.903812] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 12:36:56 (1786984616) [ 9677.426433] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9679.576453] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 12:37:00 (1786984620) [ 9683.152541] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 9714.050983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9714.207236] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9714.220469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9714.332961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9719.575850] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9729.091803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9729.200306] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9729.242260] Lustre: 311498:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9734.593469] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9757.371680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9757.767287] Lustre: Denying initial registration attempt from nid 192.168.202.137@tcp, specified as failover [ 9757.781547] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9757.792245] LustreError: 312951:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9757.803063] LustreError: 312951:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9757.815581] LustreError: 312951:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9757.983308] Lustre: server umount lustre-OST0000 complete [ 9757.987213] LustreError: 312951:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9768.930226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9768.965211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9768.971831] Lustre: Skipped 1 previous similar message [ 9769.013778] Lustre: Skipped 1 previous similar message [ 9779.172817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9779.180859] Lustre: Skipped 3 previous similar messages [ 9782.239788] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9782.646065] Lustre: server umount lustre-MDT0000 complete [ 9784.288500] LustreError: 312665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9784.298922] LustreError: 312665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9786.809032] LustreError: 311490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786984728 with bad export cookie 8799353089258752412 [ 9786.813962] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9786.817886] LustreError: 311490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9787.138932] Lustre: server umount lustre-MDT0001 complete [ 9801.184591] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 9803.901710] Key type lgssc unregistered [ 9804.173694] LNet: 314078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9804.183297] LNetError: 314078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9804.198655] LNet: Removed LNI 192.168.202.137@tcp [ 9805.178241] Key type .llcrypt unregistered [ 9805.186229] Key type ._llcrypt unregistered [ 9821.051091] Key type ._llcrypt registered [ 9821.071660] Key type .llcrypt registered [ 9821.293738] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 9822.901090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9823.104709] alg: No test for adler32 (adler32-zlib) [ 9824.728371] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 9824.911841] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 9826.527166] Key type lgssc registered [ 9827.574687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9880.596645] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 9891.732427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9891.767625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9892.995114] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9893.021148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9893.191334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9899.282254] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9908.235860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9908.358878] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9908.417581] Lustre: 316847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9908.616722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9912.720852] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9921.824909] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9927.712698] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9948.214311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9948.484084] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9948.686045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9954.314412] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9962.112210] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9965.540949] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9965.543946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9965.550325] LustreError: Skipped 1 previous similar message [ 9965.551106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9965.589861] Lustre: Skipped 1 previous similar message [ 9970.695967] Lustre: server umount lustre-OST0000 complete [ 9975.265095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9975.268403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9975.283314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9975.287713] Lustre: Skipped 1 previous similar message [ 9981.005303] Lustre: server umount lustre-MDT0000 complete [ 9985.135055] LustreError: 316838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786984927 with bad export cookie 15232304628430118707 [ 9985.135994] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9985.149040] LustreError: 316838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9985.465921] Lustre: server umount lustre-MDT0001 complete [ 9997.259690] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [10000.186815] Key type lgssc unregistered [10000.532229] LNet: 320420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10000.538757] LNetError: 320420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10000.550553] LNet: Removed LNI 192.168.202.137@tcp [10001.618551] Key type .llcrypt unregistered [10001.620456] Key type ._llcrypt unregistered [10033.302625] Key type ._llcrypt registered [10033.304713] Key type .llcrypt registered [10033.434060] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [10035.243531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10035.440893] alg: No test for adler32 (adler32-zlib) [10036.685674] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [10037.042599] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [10038.727231] Key type lgssc registered [10040.162375] Lustre: Echo OBD driver; http://www.lustre.org/ [10048.186336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10048.205886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10049.615922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10054.978931] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10059.749273] LustreError: 321837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10059.773748] LustreError: 321837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10064.224022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10064.616714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10068.602988] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10076.909196] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10083.562238] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10093.743716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10094.115946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10100.775697] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10109.308847] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10119.650305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10119.655083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10119.659826] Lustre: Skipped 1 previous similar message [10119.683967] Lustre: Skipped 2 previous similar messages [10121.698696] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.37@tcp (stopping) [10124.016936] Lustre: server umount lustre-MDT0000 complete [10124.786022] LustreError: 321837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10124.800032] LustreError: 321837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10126.759804] LustreError: 321833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10128.300774] LustreError: 321816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786985070 with bad export cookie 6345208307228736846 [10128.311728] LustreError: 321816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10128.314842] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10129.896414] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10129.897872] LustreError: 321831:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10129.914796] Lustre: Skipped 1 previous similar message [10129.949284] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10133.921331] LustreError: 324324:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10133.921563] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10133.928453] LustreError: 324324:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10133.940610] Lustre: Skipped 1 previous similar message [10134.839880] Lustre: server umount lustre-MDT0001 complete [10142.763763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10153.042107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10153.452682] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x580eb5564a3a1b31 [10153.472026] Lustre: MGC192.168.202.137@tcp: Connection restored to 0@lo (at 0@lo) [10153.923279] LustreError: 323671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10153.966801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10154.020058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10159.631546] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10163.178389] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10169.955167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10170.478327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10175.471805] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10175.814994] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10184.941818] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10186.796983] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10194.242502] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10196.844307] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10205.151928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10205.157969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10205.181131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10206.183000] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10216.418851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10216.421386] Lustre: Skipped 4 previous similar messages [10217.951161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10218.170535] Lustre: server umount lustre-OST0000 complete [10226.657306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10226.659614] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10226.681158] Lustre: Skipped 2 previous similar messages [10236.383365] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10236.662477] Lustre: server umount lustre-MDT0000 complete [10236.902028] LustreError: 327155:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10236.927948] LustreError: 327155:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10240.885132] LustreError: 321817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786985183 with bad export cookie 6345208307228744497 [10240.886916] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10240.911740] LustreError: 321817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10241.425574] Lustre: server umount lustre-MDT0001 complete [10255.860700] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [10259.108659] Key type lgssc unregistered [10259.385676] LNet: 328113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10259.400158] LNetError: 328113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10259.417852] LNet: Removed LNI 192.168.202.137@tcp [10260.420670] Key type .llcrypt unregistered [10260.422574] Key type ._llcrypt unregistered [10288.370576] Key type ._llcrypt registered [10288.373630] Key type .llcrypt registered [10288.506920] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [10289.612575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10289.620569] alg: No test for adler32 (adler32-zlib) [10290.644898] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [10290.852243] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [10292.504879] Key type lgssc registered [10293.651410] Lustre: Echo OBD driver; http://www.lustre.org/ [10300.797410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10300.827749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10302.423945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10307.357909] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10312.679081] LustreError: 329531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10312.697694] LustreError: 329531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10316.251400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10316.695683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10322.471071] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10330.117471] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10336.523580] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10341.868099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10341.878283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10346.977365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10346.980736] Lustre: Skipped 2 previous similar messages [10352.100282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10352.104627] Lustre: Skipped 1 previous similar message [10354.143522] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10354.490178] Lustre: server umount lustre-MDT0000 complete [10357.224295] LustreError: 329532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10357.243687] LustreError: 329532:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10358.812899] LustreError: 329511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786985300 with bad export cookie 297203485505078530 [10358.819538] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10358.821099] LustreError: 329511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10359.327618] Lustre: server umount lustre-MDT0001 complete [10374.930163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10375.238457] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10375.262905] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10375.512513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10379.880327] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10390.053536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10390.135717] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10390.174374] Lustre: 332121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10390.554916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10397.155800] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10406.140251] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10412.850760] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10423.826774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10424.130039] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10424.164865] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10424.186274] LustreError: 334007:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10424.192926] LustreError: 334007:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10424.201713] LustreError: 334007:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10424.313517] Lustre: server umount lustre-OST0000 complete [10424.317620] LustreError: 334007:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10436.147724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10436.407756] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10436.637244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10442.914856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10443.668107] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10452.458322] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10462.952046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10463.030807] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10463.096536] Lustre: lustre-OST0001: new disk, initializing [10463.099831] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10463.105630] Lustre: 335553:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10463.199956] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10471.224158] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10472.593483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10472.599839] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10472.699337] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10480.068466] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10488.291623] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10488.295492] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10488.310456] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10488.314267] Lustre: Skipped 2 previous similar messages [10488.326509] Lustre: Skipped 1 previous similar message [10489.458476] Lustre: server umount lustre-OST0001 complete [10493.422618] LustreError: 334477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10493.444386] LustreError: 334477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10494.433365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10494.441569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10494.452460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10498.531588] LustreError: 334477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10498.535776] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10498.574281] LustreError: 334477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10500.611553] Lustre: server umount lustre-OST0000 complete [10508.256445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10508.266076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10508.284137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10508.288020] Lustre: Skipped 2 previous similar messages [10510.941632] Lustre: server umount lustre-MDT0000 complete [10513.890835] LustreError: 332994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10513.918634] LustreError: 332994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10515.405545] LustreError: 332859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786985457 with bad export cookie 297203485505079083 [10515.425398] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10515.886558] Lustre: server umount lustre-MDT0001 complete [10528.692611] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [10531.907824] Key type lgssc unregistered [10532.225847] LNet: 337514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10532.234470] LNetError: 337514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10532.262314] LNet: Removed LNI 192.168.202.137@tcp [10533.359502] Key type .llcrypt unregistered [10533.364108] Key type ._llcrypt unregistered [10613.299692] Key type ._llcrypt registered [10613.304337] Key type .llcrypt registered [10613.435191] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [10632.656557] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [10634.140291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10634.188822] alg: No test for adler32 (adler32-zlib) [10635.311577] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [10635.544368] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [10637.247746] Key type lgssc registered [10638.653248] Lustre: Echo OBD driver; http://www.lustre.org/ [10681.168295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10681.238111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10682.473683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10682.514527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10682.633785] Lustre: lustre-MDT0000: new disk, initializing [10682.712557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10682.756810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10687.295290] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10698.857146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10698.974953] Lustre: 342452:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10699.058427] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10699.061504] Lustre: Skipped 1 previous similar message [10699.209832] Lustre: lustre-MDT0001: new disk, initializing [10699.344979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10699.390321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10699.405984] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10704.125858] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10714.270329] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10720.287451] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10727.692320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10727.994492] Lustre: lustre-OST0000: new disk, initializing [10728.006208] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10728.020583] Lustre: 344419:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10728.148633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10734.634170] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10736.738133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10736.747106] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10736.840654] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10744.776666] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10749.882732] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10750.143329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10755.618635] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10755.849475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10762.724402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10762.729981] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10762.739812] Lustre: Skipped 1 previous similar message [10762.741058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10764.705821] Lustre: server umount lustre-OST0000 complete [10770.914129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10770.923728] LustreError: Skipped 1 previous similar message [10770.932826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10770.952220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10770.961278] Lustre: Skipped 1 previous similar message [10772.964200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10772.966022] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10777.368835] Lustre: server umount lustre-MDT0000 complete [10780.979175] LustreError: 342444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786985723 with bad export cookie 15839844146764711627 [10780.993553] LustreError: 342444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10780.996972] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10781.582745] Lustre: server umount lustre-MDT0001 complete [10792.248028] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:55:32 (1786985732) [10801.232901] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:55:41 (1786985741) [10810.496644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10810.921196] LustreError: 347178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10810.939419] LustreError: 347178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10810.991863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10815.410261] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10816.499102] LustreError: 347179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10821.614959] LustreError: 347178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10824.426402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10824.806472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10830.406810] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10838.593273] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10845.993905] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10854.600744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10855.233724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10862.824523] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10871.812515] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10881.983908] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10884.732187] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10887.027673] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10889.539571] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10892.049509] Lustre: Failing over lustre-MDT0000 [10892.522263] Lustre: server umount lustre-MDT0000 complete [10895.319195] LustreError: 347175:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10895.330261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10895.353035] LustreError: 347175:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10895.379221] Lustre: Skipped 2 previous similar messages [10900.395059] LustreError: 347173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10910.630536] LustreError: 347175:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10910.648987] LustreError: 347175:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10911.711388] Lustre: 340672:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786985837/real 1786985837] req@000000004a4123fa x1873790188464128/t0(0) o400->MGC192.168.202.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786985853 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10911.726475] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10912.621332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10920.930994] LustreError: 340671:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000082a41917 x1873790188469632/t0(0) o250->MGC192.168.202.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10921.325530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10921.408632] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10925.991159] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10926.604383] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10926.664608] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10926.737307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10927.360478] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10940.501531] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10942.701616] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10946.527351] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10948.669163] Lustre: Failing over lustre-MDT0000 [10949.150318] Lustre: server umount lustre-MDT0000 complete [10951.618057] LustreError: 349295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10951.636604] LustreError: 349295:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [10952.163659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10952.166834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10952.187928] Lustre: Skipped 2 previous similar messages [10968.543151] Lustre: 340675:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786985894/real 1786985894] req@00000000ac2e836f x1873790188487040/t0(0) o400->MGC192.168.202.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786985910 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10968.568172] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10968.837409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10978.787299] LustreError: 340671:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b7931a90 x1873790188493184/t0(0) o250->MGC192.168.202.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10979.145709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10979.213801] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10982.307074] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10984.428030] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10984.432228] Lustre: Skipped 2 previous similar messages [10984.469314] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10984.520990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10984.563595] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10995.640719] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10997.512465] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11004.896801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11004.901937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11004.913527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11004.925182] Lustre: Skipped 1 previous similar message [11004.940186] Lustre: Skipped 3 previous similar messages [11010.029178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11010.038829] Lustre: Skipped 2 previous similar messages [11011.133212] Lustre: server umount lustre-MDT0000 complete [11014.854691] LustreError: 347159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786985957 with bad export cookie 15839844146764714343 [11014.870551] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11015.141214] LustreError: 347173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11015.155588] LustreError: 347173:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [11015.292512] Lustre: server umount lustre-MDT0001 complete [11021.963866] Lustre: server umount lustre-OST0000 complete [11037.155941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11041.853757] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11050.603268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11050.940344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11050.944243] Lustre: Skipped 1 previous similar message [11056.058322] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11064.980638] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11071.394587] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11078.679526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11082.049770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11085.536398] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11093.014984] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11104.745639] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11104.755160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11104.764141] Lustre: Skipped 2 previous similar messages [11104.789385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11110.133231] Lustre: server umount lustre-OST0000 complete [11122.144435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11122.153058] Lustre: Skipped 5 previous similar messages [11128.287190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11128.493949] Lustre: server umount lustre-MDT0000 complete [11132.394779] LustreError: 354247:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11132.419602] LustreError: 354247:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11133.435759] LustreError: 355364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786986075 with bad export cookie 15839844146764715085 [11133.439213] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11133.449399] LustreError: 355364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11133.915678] Lustre: server umount lustre-MDT0001 complete [11147.749316] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [11151.222633] Key type lgssc unregistered [11151.619257] LNet: 357067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11151.627346] LNetError: 357067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11151.652142] LNet: Removed LNI 192.168.202.137@tcp [11152.542707] Key type .llcrypt unregistered [11152.546800] Key type ._llcrypt unregistered [11181.741416] Key type ._llcrypt registered [11181.749166] Key type .llcrypt registered [11181.834989] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [11182.935807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11183.003784] alg: No test for adler32 (adler32-zlib) [11184.094680] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [11184.319244] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [11185.967196] Key type lgssc registered [11187.053402] Lustre: Echo OBD driver; http://www.lustre.org/ [11194.204881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11194.229114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11195.669447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11200.181578] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11205.089748] LustreError: 358585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11205.127195] LustreError: 358585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11209.726938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11210.257810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11215.381512] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11222.976162] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11229.087346] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11236.198652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11236.826714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11239.932621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11243.408817] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11251.429672] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11266.446739] Lustre: server umount lustre-MDT0000 complete [11266.528159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11266.531520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11266.547501] LustreError: 358586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11269.899074] LustreError: 358566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786986212 with bad export cookie 17069679526755107218 [11269.943956] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11270.254293] Lustre: server umount lustre-MDT0001 complete [11294.928319] Lustre: server umount lustre-OST0000 complete [11310.274570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11310.806848] LustreError: 362321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11310.882502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11316.031291] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11316.197696] LustreError: 362322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11320.310683] LustreError: 362321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11325.728886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11326.232257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11332.066551] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11341.476811] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11349.820305] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11359.018759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11359.325320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11366.132873] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11373.965235] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11380.864135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11398.112210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11398.119921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11398.131571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11401.700158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11401.704211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11401.737243] Lustre: Skipped 1 previous similar message [11403.973962] Lustre: server umount lustre-MDT0000 complete [11406.817357] LustreError: 362321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11406.843273] LustreError: 362321:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11407.662171] LustreError: 362301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786986349 with bad export cookie 17069679526755108338 [11407.683127] LustreError: 362301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11407.685105] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11408.127766] Lustre: server umount lustre-MDT0001 complete [11423.066308] Lustre: server umount lustre-OST0000 complete [11441.879392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11442.310792] LustreError: 366255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11442.373946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11447.669276] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11457.175034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11462.868313] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11471.189695] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11478.248438] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11486.570847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11486.799624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11486.803818] Lustre: Skipped 1 previous similar message [11487.861340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11493.773101] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11502.831732] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11512.803570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11512.822817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11512.847153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11512.849226] Lustre: Skipped 2 previous similar messages [11515.606454] Lustre: server umount lustre-OST0000 complete [11523.040153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11523.045527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11523.055640] Lustre: Skipped 2 previous similar messages [11523.067187] Lustre: Skipped 2 previous similar messages [11526.227501] Lustre: server umount lustre-MDT0000 complete [11528.165500] LustreError: 366255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11528.188377] LustreError: 366255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11530.165981] LustreError: 369038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786986472 with bad export cookie 17069679526755109360 [11530.173338] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11530.178460] LustreError: 369038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11530.548773] Lustre: server umount lustre-MDT0001 complete [11543.905179] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [11547.077602] Key type lgssc unregistered [11547.442457] LNet: 369792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11547.446473] LNetError: 369792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11547.470935] LNet: Removed LNI 192.168.202.137@tcp [11548.535708] Key type .llcrypt unregistered [11548.538052] Key type ._llcrypt unregistered [11576.592921] Key type ._llcrypt registered [11576.596935] Key type .llcrypt registered [11576.680843] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [11578.160962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11578.310286] alg: No test for adler32 (adler32-zlib) [11579.411861] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [11579.762541] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [11581.489407] Key type lgssc registered [11582.833754] Lustre: Echo OBD driver; http://www.lustre.org/ [11591.386618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11591.453941] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11591.466270] LustreError: 371234:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11591.487512] LustreError: 371234:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11591.493925] LustreError: 371234:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11591.505954] LustreError: 371234:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11591.513921] LustreError: 371234:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11598.986177] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11598.991370] LustreError: 371632:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11599.017762] LustreError: 371632:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11599.029576] LustreError: 371632:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11599.040221] LustreError: 371632:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11599.046556] LustreError: 371632:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11620.165232] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [11623.220441] Key type lgssc unregistered [11623.554484] LNet: 372640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11623.567096] LNetError: 372640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11623.593096] LNet: Removed LNI 192.168.202.137@tcp [11624.454290] Key type .llcrypt unregistered [11624.456703] Key type ._llcrypt unregistered [11644.010944] Key type ._llcrypt registered [11644.013318] Key type .llcrypt registered [11644.177865] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [11660.768592] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [11661.958400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11662.183802] alg: No test for adler32 (adler32-zlib) [11663.236869] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [11663.609744] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [11665.343148] Key type lgssc registered [11666.570417] Lustre: Echo OBD driver; http://www.lustre.org/ [11704.205522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11704.250332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11705.485307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11705.516455] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11705.588748] Lustre: lustre-MDT0000: new disk, initializing [11705.716745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11705.734492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11711.240600] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11722.188950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11722.340296] Lustre: 375810:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11722.427261] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11722.429618] Lustre: Skipped 1 previous similar message [11722.491422] Lustre: lustre-MDT0001: new disk, initializing [11722.575643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11722.613624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11722.625265] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11727.567220] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11738.621118] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11745.008766] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11752.024537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11752.249519] Lustre: lustre-OST0000: new disk, initializing [11752.253913] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11752.259703] Lustre: 377777:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11752.340832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11758.590325] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11761.851207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11761.872314] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11762.094383] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11771.067315] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11777.022754] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11777.270835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11784.064884] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11784.431826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11787.750774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11787.752697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11787.773196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11787.783563] Lustre: Skipped 1 previous similar message [11787.811298] Lustre: Skipped 1 previous similar message [11792.937331] Lustre: server umount lustre-OST0000 complete [11797.995939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11798.005791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11798.009781] Lustre: Skipped 1 previous similar message [11798.024303] Lustre: Skipped 1 previous similar message [11803.104358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11803.113671] Lustre: Skipped 1 previous similar message [11803.526994] Lustre: server umount lustre-MDT0000 complete [11807.071395] LustreError: 375800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786986749 with bad export cookie 5545539198764275102 [11807.073300] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11807.084173] LustreError: 375800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11807.514359] Lustre: server umount lustre-MDT0001 complete [11817.764382] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11820.128030] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:12:40 (1786986760) [11828.058415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11828.585596] LustreError: 380238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11828.606424] LustreError: 380238:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11828.705038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11833.839334] LustreError: 380239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11833.870830] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11837.936780] LustreError: 380239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11843.054188] LustreError: 380238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11843.972400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11844.381934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11849.602524] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11857.883413] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11864.658371] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11872.700901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11873.155331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11880.079264] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11887.999482] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11895.346337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11895.639964] Lustre: lustre-OST0001: new disk, initializing [11895.645365] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11895.652202] Lustre: 383075:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11895.751585] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11901.928931] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11905.785970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11905.801114] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11905.942431] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11913.590593] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11921.379039] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11921.400594] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11921.402445] Lustre: Skipped 1 previous similar message [11925.696577] Lustre: server umount lustre-OST0001 complete [11926.522828] LustreError: 382060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11926.550262] LustreError: 382060:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11936.742892] LustreError: 382062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11936.772687] LustreError: 382062:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11945.967112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11945.973962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11945.975944] LustreError: Skipped 2 previous similar messages [11946.005110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11946.005121] Lustre: Skipped 2 previous similar messages [11951.783124] Lustre: server umount lustre-OST0000 complete [11957.219029] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11957.223385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11957.230955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11957.230964] Lustre: Skipped 3 previous similar messages [11957.263372] Lustre: Skipped 2 previous similar messages [11962.708130] Lustre: server umount lustre-MDT0000 complete [11967.358215] LustreError: 383086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786986909 with bad export cookie 5545539198764275984 [11967.359764] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11967.377668] LustreError: 383086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11974.123229] Lustre: server umount lustre-MDT0001 complete [11988.478576] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [11991.881351] Key type lgssc unregistered [11992.145413] LNet: 385070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11992.152163] LNetError: 385070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11992.173793] LNet: Removed LNI 192.168.202.137@tcp [11993.196695] Key type .llcrypt unregistered [11993.198950] Key type ._llcrypt unregistered [12055.519233] Key type ._llcrypt registered [12055.520829] Key type .llcrypt registered [12055.638206] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [12057.940379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12058.000123] alg: No test for adler32 (adler32-zlib) [12059.435298] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [12059.844790] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [12061.751337] Key type lgssc registered [12063.623832] Lustre: Echo OBD driver; http://www.lustre.org/ [12074.775940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12074.796180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12076.200054] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12076.229327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12076.475130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12080.422489] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12089.903524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12090.023552] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12090.066734] Lustre: 387875:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12090.454346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12095.042669] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12102.989758] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12110.120181] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12113.381035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12113.414225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12113.428860] Lustre: Skipped 1 previous similar message [12118.496910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12118.504911] Lustre: Skipped 1 previous similar message [12118.703469] Lustre: server umount lustre-MDT0000 complete [12123.297691] LustreError: 387868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786987065 with bad export cookie 14746038338831934636 [12123.304136] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12123.323037] LustreError: 387868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12123.703103] Lustre: server umount lustre-MDT0001 complete [12135.431698] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:17:55 (1786987075) [12143.182312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12143.894789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12148.738955] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12149.224221] LustreError: 390389:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12149.251818] LustreError: 390389:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12154.341414] LustreError: 390390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12158.912291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12159.288393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12164.157770] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12173.949213] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12180.934426] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12188.461334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12188.679826] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12188.946882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12195.728356] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12203.458583] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12211.938868] Lustre: Permanently deactivating lustre-OST0000 [12211.955148] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12220.437249] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12222.621279] LustreError: 393377:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12222.646653] LustreError: 393377:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12222.667124] LustreError: 393377:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12227.239691] LustreError: 393425:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12227.245195] LustreError: 393425:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12235.922915] Lustre: server umount lustre-OST0000 complete [12238.869526] LustreError: 393676:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12238.876292] LustreError: 393676:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12241.376415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12241.382061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12241.399191] Lustre: Skipped 1 previous similar message [12241.412402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12243.939106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12243.944279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12249.056086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12249.064667] Lustre: Skipped 2 previous similar messages [12255.199307] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12255.420265] Lustre: server umount lustre-MDT0000 complete [12259.299526] LustreError: 390385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12259.313573] LustreError: 390385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12259.368722] LustreError: 393774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786987201 with bad export cookie 14746038338831935217 [12259.375486] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12259.377557] LustreError: 393774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12269.664264] Lustre: server umount lustre-MDT0001 complete [12278.705285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12284.067586] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12286.716496] LustreError: 394837:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12293.477951] Lustre: 395175:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.137@tcp [12296.008636] Lustre: server umount MGS complete [12309.447300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12309.810137] LustreError: 395899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12309.832250] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12309.844580] Lustre: Skipped 1 previous similar message [12309.924475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12315.121436] LustreError: 395900:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12315.679819] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12325.344925] LustreError: 395900:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12325.357348] LustreError: 395900:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12327.731682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12328.394349] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12328.598282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12334.071376] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12342.080196] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12349.068288] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12351.125292] Lustre: Permanently reactivating lustre-OST0000 [12351.141204] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12351.148547] Lustre: Skipped 2 previous similar messages [12380.127176] LustreError: 397723:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12387.707937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12388.146968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12394.519560] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12403.577226] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12409.827989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12409.834988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12409.853979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12409.859675] Lustre: Skipped 2 previous similar messages [12410.859832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12411.733301] Lustre: server umount lustre-OST0000 complete [12415.976592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12415.999301] Lustre: Skipped 1 previous similar message [12420.577550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12421.981951] Lustre: server umount lustre-MDT0000 complete [12426.136378] LustreError: 395881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786987368 with bad export cookie 14746038338831936267 [12426.145738] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12426.149443] LustreError: 395881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12426.410537] Lustre: server umount lustre-MDT0001 complete [12438.613378] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [12441.345783] Key type lgssc unregistered [12441.634635] LNet: 400317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12441.644711] LNetError: 400317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12441.661903] LNet: Removed LNI 192.168.202.137@tcp [12442.445650] Key type .llcrypt unregistered [12442.449293] Key type ._llcrypt unregistered [12463.124521] Key type ._llcrypt registered [12463.134445] Key type .llcrypt registered [12463.258447] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [12482.648958] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [12484.041946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12484.149574] alg: No test for adler32 (adler32-zlib) [12485.255846] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [12485.557129] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [12487.231456] Key type lgssc registered [12488.603647] Lustre: Echo OBD driver; http://www.lustre.org/ [12524.250613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12524.285202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12525.697704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12525.741925] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12525.873390] Lustre: lustre-MDT0000: new disk, initializing [12525.965125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12525.976939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12530.359130] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12544.290536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12544.418993] Lustre: 403479:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12544.460995] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12544.466952] Lustre: Skipped 1 previous similar message [12544.569086] Lustre: lustre-MDT0001: new disk, initializing [12544.713615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12544.759942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12544.777626] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12550.783784] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12563.443276] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12570.947055] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12578.346190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12578.551258] Lustre: lustre-OST0000: new disk, initializing [12578.554604] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12578.561976] Lustre: 405448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12578.652156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12584.544325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12584.567892] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12584.669613] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12585.288828] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12595.228518] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12600.235351] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12600.475680] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12605.516563] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12605.730378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12610.530321] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12610.533117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12610.547477] LustreError: Skipped 1 previous similar message [12610.548495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12610.569154] Lustre: Skipped 1 previous similar message [12615.668790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12615.676985] Lustre: Skipped 2 previous similar messages [12620.770389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12620.778532] Lustre: Skipped 1 previous similar message [12622.304259] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12622.544300] Lustre: server umount lustre-OST0000 complete [12626.914031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12626.930667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12626.950746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12631.009829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12631.012676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12632.939265] Lustre: server umount lustre-MDT0000 complete [12637.121567] LustreError: 403470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786987579 with bad export cookie 14182284188214414251 [12637.127513] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12637.129132] LustreError: 403470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12637.489819] Lustre: server umount lustre-MDT0001 complete [12647.135712] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12649.458966] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:26:29 (1786987589) [12657.311242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12657.881741] LustreError: 407912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12657.902338] LustreError: 407912:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12657.965894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12663.286525] LustreError: 407912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12664.130401] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12668.388138] LustreError: 407913:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12674.530739] LustreError: 407912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12674.556959] LustreError: 407912:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12675.339769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12675.594829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12681.363699] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12691.861893] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12702.061509] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12710.802830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12711.273739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12717.776443] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12726.108534] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12728.495916] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12731.869571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12740.066391] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12740.078333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12740.086332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12740.086340] Lustre: Skipped 1 previous similar message [12740.350239] Lustre: server umount lustre-OST0000 complete [12745.212186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12745.232205] Lustre: Skipped 2 previous similar messages [12751.358039] Lustre: server umount lustre-MDT0000 complete [12755.078195] LustreError: 407893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786987697 with bad export cookie 14182284188214415133 [12755.092032] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12755.101022] LustreError: 407893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12755.630657] Lustre: server umount lustre-MDT0001 complete [12769.539529] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [12772.912929] Key type lgssc unregistered [12773.230767] LNet: 411641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12773.256033] LNetError: 411641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12773.270949] LNet: Removed LNI 192.168.202.137@tcp [12773.978238] Key type .llcrypt unregistered [12773.981523] Key type ._llcrypt unregistered [12800.861974] Key type ._llcrypt registered [12800.863653] Key type .llcrypt registered [12801.057224] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [12801.362236] Key type .llcrypt unregistered [12801.366527] Key type ._llcrypt unregistered [12822.009432] Key type ._llcrypt registered [12822.011579] Key type .llcrypt registered [12822.124973] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [12823.580679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12823.621686] alg: No test for adler32 (adler32-zlib) [12824.661721] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [12824.962781] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [12826.671853] Key type lgssc registered [12827.540354] Lustre: Echo OBD driver; http://www.lustre.org/ [12834.843894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12834.869268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12836.289213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12841.460716] LustreError: 413859:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12841.500835] LustreError: 413859:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12841.641653] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12846.569214] LustreError: 413860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12851.684839] LustreError: 413860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12852.901219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12853.304810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12858.337925] LustreError: 414604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12858.350193] LustreError: 414604:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12861.446975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12861.616045] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12866.547597] LustreError: 413860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12866.572000] LustreError: 413860:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12874.705842] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12876.778885] LustreError: 413859:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12876.800738] LustreError: 413859:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12883.871876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12884.418958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12889.363494] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12899.809799] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12899.816193] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12899.826205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12899.829637] Lustre: Skipped 1 previous similar message [12899.853036] Lustre: Skipped 1 previous similar message [12904.936330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12904.942862] Lustre: Skipped 1 previous similar message [12905.720798] Lustre: server umount lustre-OST0000 complete [12910.048992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12910.055678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12910.070535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12910.086221] Lustre: Skipped 1 previous similar message [12912.382769] Lustre: server umount lustre-MDT0000 complete [12915.168992] LustreError: 415866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12915.188588] LustreError: 415866:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12917.398701] LustreError: 413839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786987859 with bad export cookie 8084402488916908998 [12917.403572] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12917.418834] LustreError: 413839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12917.809921] Lustre: server umount lustre-MDT0001 complete [12932.055484] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [12935.139393] Key type lgssc unregistered [12935.636165] LNet: 417012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12935.649548] LNetError: 417012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12935.668765] LNet: Removed LNI 192.168.202.137@tcp [12936.609091] Key type .llcrypt unregistered [12936.618721] Key type ._llcrypt unregistered [12973.177560] Key type ._llcrypt registered [12973.179376] Key type .llcrypt registered [12973.269378] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [12974.510748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12974.599061] alg: No test for adler32 (adler32-zlib) [12975.765250] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [12976.171197] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [12977.919204] Key type lgssc registered [12979.520179] Lustre: Echo OBD driver; http://www.lustre.org/ [12988.136716] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12988.147930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13003.807822] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13008.927442] LustreError: 418406:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.137@tcp: failed processing log, type 4: rc = -110 [13035.551170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13044.522733] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13055.997183] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13064.633506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13065.346096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13065.414730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13070.565194] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13070.817583] LustreError: 418432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13070.838393] LustreError: 418432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13071.843353] LustreError: 418431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13071.861612] LustreError: 418431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13076.977039] LustreError: 418432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13077.002261] LustreError: 418432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13081.005479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13081.435668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13086.152605] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13095.287092] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13103.186040] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13111.776384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13111.786852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13111.808260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13112.315087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13112.320649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13112.394165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13112.397073] Lustre: Skipped 1 previous similar message [13117.135973] Lustre: server umount lustre-OST0000 complete [13122.529300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13122.533070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13122.534584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13122.566989] Lustre: Skipped 1 previous similar message [13127.653534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13127.665258] Lustre: Skipped 2 previous similar messages [13128.140429] Lustre: server umount lustre-MDT0000 complete [13132.220402] LustreError: 418414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786988074 with bad export cookie 9853834207075619785 [13132.232150] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13132.615360] Lustre: server umount lustre-MDT0001 complete [13147.118767] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [13150.557265] Key type lgssc unregistered [13150.753770] LNet: 421983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13150.761058] LNetError: 421983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13150.817491] LNet: Removed LNI 192.168.202.137@tcp [13152.011358] Key type .llcrypt unregistered [13152.013746] Key type ._llcrypt unregistered [13190.974795] Key type ._llcrypt registered [13190.977267] Key type .llcrypt registered [13191.169692] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [13192.346115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13192.466718] alg: No test for adler32 (adler32-zlib) [13193.652920] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [13194.060417] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [13195.919283] Key type lgssc registered [13197.442270] Lustre: Echo OBD driver; http://www.lustre.org/ [13206.113612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13206.132446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13207.612101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13213.092182] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13217.763371] LustreError: 423406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13217.789247] LustreError: 423406:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13222.888777] LustreError: 423405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13223.709332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13224.196743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13228.915106] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13237.376924] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13244.720897] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13252.035649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13252.276594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13258.079701] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13266.525788] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13277.892341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13283.301052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13283.302087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13283.316302] Lustre: Skipped 2 previous similar messages [13283.331678] Lustre: Skipped 2 previous similar messages [13288.083066] Lustre: server umount lustre-MDT0000 complete [13288.419585] LustreError: 424133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13288.443263] LustreError: 424133:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13291.192156] LustreError: 425249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.37@tcp arrived at 1786988233 with bad export cookie 6503991647952401294 [13291.205858] LustreError: 425249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13293.040052] LustreError: 423387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786988235 with bad export cookie 6503991647952400636 [13293.048804] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13293.052339] LustreError: 423387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13293.536459] LustreError: 423405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13293.547852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13293.565402] LustreError: 423405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13293.591244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13293.598199] Lustre: Skipped 1 previous similar message [13298.658107] LustreError: 423406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13298.658988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13298.679398] LustreError: 423406:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13299.314312] Lustre: server umount lustre-OST0000 complete [13307.121952] Lustre: server umount lustre-MDT0001 complete [13319.660056] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [13322.779685] Key type lgssc unregistered [13323.082832] LNet: 427115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13323.099099] LNetError: 427115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13323.118203] LNet: Removed LNI 192.168.202.137@tcp [13324.004924] Key type .llcrypt unregistered [13324.006948] Key type ._llcrypt unregistered [13351.458692] Key type ._llcrypt registered [13351.468928] Key type .llcrypt registered [13351.618585] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [13352.954770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13353.070851] alg: No test for adler32 (adler32-zlib) [13354.134862] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [13354.391521] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [13356.087178] Key type lgssc registered [13357.661143] Lustre: Echo OBD driver; http://www.lustre.org/ [13367.678603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13367.714341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13369.646379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13375.043148] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13379.045106] LustreError: 428533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13379.072053] LustreError: 428533:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13384.175239] LustreError: 428534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13386.669097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13387.538519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13393.705148] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13405.174680] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13413.464483] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13420.553470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13420.955593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13428.862140] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13438.188436] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13439.468281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:193) [13444.065214] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13444.071133] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13444.112396] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13444.581035] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13446.614132] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.37@tcp (stopping) [13446.627672] Lustre: Skipped 1 previous similar message [13449.701177] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13449.715223] Lustre: Skipped 1 previous similar message [13449.963975] Lustre: server umount lustre-MDT0001 complete [13451.690655] LustreError: 428528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13454.819874] LustreError: 428530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13454.857883] LustreError: 428530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13459.939869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13459.957829] LustreError: 428533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13459.982564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13464.813893] Lustre: server umount lustre-OST0000 complete [13469.898205] Lustre: server umount lustre-MDT0000 complete [13485.173814] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [13488.405420] Key type lgssc unregistered [13488.771061] LNet: 432223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13488.777839] LNetError: 432223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13488.794262] LNet: Removed LNI 192.168.202.137@tcp [13489.865632] Key type .llcrypt unregistered [13489.868544] Key type ._llcrypt unregistered [13525.739324] Key type ._llcrypt registered [13525.741471] Key type .llcrypt registered [13525.848508] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [13543.848949] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [13546.557704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13546.610000] alg: No test for adler32 (adler32-zlib) [13547.781914] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [13548.092581] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [13549.951272] Key type lgssc registered [13551.212035] Lustre: Echo OBD driver; http://www.lustre.org/ [13604.847847] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [13623.348549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13623.391298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13624.725556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13624.745344] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13624.829358] Lustre: lustre-MDT0000: new disk, initializing [13624.885909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13624.900262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13629.560673] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13649.306196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13649.791961] Lustre: lustre-OST0000: new disk, initializing [13649.796759] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13649.815892] Lustre: Skipped 1 previous similar message [13649.820506] Lustre: 437289:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13649.956785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13658.728526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13658.736783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13658.777388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13660.667238] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13671.635393] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13688.309624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13688.464211] Lustre: 436268:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13688.551251] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13688.646637] Lustre: lustre-MDT0001: new disk, initializing [13688.795074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13688.851197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13688.872151] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13693.141924] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13702.614107] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13702.821854] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13706.816326] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13707.016176] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13711.842146] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13711.862717] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13711.868639] Lustre: Skipped 1 previous similar message [13715.420633] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.37@tcp (stopping) [13716.973451] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13716.984289] Lustre: Skipped 1 previous similar message [13718.142956] Lustre: server umount lustre-MDT0001 complete [13727.207606] LustreError: 436280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13727.242442] LustreError: 436280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [13732.325294] LustreError: 436279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13732.371873] LustreError: 436279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13737.444885] LustreError: 436280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13737.447361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13737.526637] Lustre: Skipped 1 previous similar message [13737.532487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13742.575225] LustreError: 436279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13742.584197] LustreError: 436279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13742.727830] Lustre: server umount lustre-OST0000 complete [13753.821053] Lustre: server umount lustre-MDT0000 complete [13774.317811] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [13778.070399] Key type lgssc unregistered [13778.435748] LNet: 441039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13778.443744] LNetError: 441039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13779.501356] LNet: Removed LNI 192.168.202.137@tcp [13780.958783] Key type .llcrypt unregistered [13780.960244] Key type ._llcrypt unregistered [13829.313831] Key type ._llcrypt registered [13829.315406] Key type .llcrypt registered [13829.399594] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [13829.708554] Key type .llcrypt unregistered [13829.711146] Key type ._llcrypt unregistered [13849.889386] Key type ._llcrypt registered [13849.895617] Key type .llcrypt registered [13850.050682] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [13850.957270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13851.072060] alg: No test for adler32 (adler32-zlib) [13852.208174] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [13852.511912] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [13854.160053] Key type lgssc registered [13855.260215] Lustre: Echo OBD driver; http://www.lustre.org/ [13886.239508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13886.284670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13887.607988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13887.643045] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13887.716801] Lustre: lustre-MDT0000: new disk, initializing [13887.788461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13887.812205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13892.452038] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13903.619852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13903.743329] Lustre: 444421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13903.789745] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13903.793447] Lustre: Skipped 1 previous similar message [13903.848945] Lustre: lustre-MDT0001: new disk, initializing [13903.923954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13903.959095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13903.981218] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13908.852861] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13919.722200] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13926.185629] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13933.991869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13934.276290] Lustre: lustre-OST0000: new disk, initializing [13934.279534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13934.284645] Lustre: 446393:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13934.371847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13941.310309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13941.317572] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13941.448742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13941.678679] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13952.664582] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13962.223710] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13962.224830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13962.250164] Lustre: Skipped 2 previous similar messages [13962.254463] Lustre: Skipped 1 previous similar message [13967.381740] Lustre: server umount lustre-MDT0000 complete [13971.428802] LustreError: 444413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786988913 with bad export cookie 12880230672377547066 [13971.448318] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13971.453059] LustreError: 444413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13971.988530] Lustre: server umount lustre-MDT0001 complete [13985.436510] Lustre: server umount lustre-OST0000 complete [13996.866979] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 13:48:57 (1786988937) [14005.170769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14005.734375] LustreError: 448421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14005.747210] LustreError: 448421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14005.835261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14010.341616] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [14010.870469] LustreError: 448422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14014.959353] LustreError: 448421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14020.068429] LustreError: 448422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14020.185473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14020.922794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14026.856154] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [14034.823701] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14041.584541] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14062.957509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14063.240904] Lustre: Found index 0 for lustre-OST0000, updating log [14063.434625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14069.959430] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [14076.784444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14078.734081] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14086.121955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14086.130595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14086.151701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14086.154390] Lustre: Skipped 1 previous similar message [14087.205556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14087.212262] Lustre: Skipped 2 previous similar messages [14090.876893] Lustre: server umount lustre-OST0000 complete [14108.642404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14108.647601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14108.663321] Lustre: Skipped 2 previous similar messages [14108.674295] Lustre: Skipped 1 previous similar message [14113.760589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14113.766434] Lustre: Skipped 1 previous similar message [14114.122300] Lustre: server umount lustre-MDT0000 complete [14114.788896] LustreError: 448421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14114.821214] LustreError: 448421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14118.735623] LustreError: 448403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786989060 with bad export cookie 12880230672377548459 [14118.742559] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14118.752292] LustreError: 448403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14119.200810] Lustre: server umount lustre-MDT0001 complete [14128.993527] LNet: 1 local NIs in recovery (showing 1): 192.168.202.137@tcp [14130.406412] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 13:51:10 (1786989070) [14760.190659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14760.755190] LustreError: 454845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14760.789341] LustreError: 454845:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14760.894443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14766.058690] LustreError: 454846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14766.321266] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [14770.151161] LustreError: 454845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14776.290825] LustreError: 454845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14776.314019] LustreError: 454845:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14777.263031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14777.799056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14783.886629] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [14795.019797] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14805.009624] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14819.170418] LNet: 1 local NIs in recovery (showing 1): 192.168.202.137@tcp [14828.611391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14828.837764] Lustre: Found index 0 for lustre-OST0000, updating log [14829.170626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14834.316647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [14836.535338] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [14844.159307] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14850.017105] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14850.035259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14850.045743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14852.928108] Lustre: server umount lustre-OST0000 complete [14857.762340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14857.764864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14857.774503] Lustre: Skipped 1 previous similar message [14857.786384] Lustre: Skipped 1 previous similar message [14859.810070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14859.815843] Lustre: Skipped 3 previous similar messages [14863.905368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14863.919547] Lustre: Skipped 7 previous similar messages [14871.008128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14871.271200] Lustre: server umount lustre-MDT0000 complete [14872.097579] LustreError: 454846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14872.106735] LustreError: 454846:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14875.790634] LustreError: 454828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786989817 with bad export cookie 12880230672377549544 [14875.794683] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14875.805312] LustreError: 454828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14876.203865] Lustre: server umount lustre-MDT0001 complete [14879.583766] LNet: 1 local NIs in recovery (showing 1): 192.168.202.137@tcp [14908.984712] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [14921.993873] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [14966.904055] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 14:05:07 (1786989907) [14992.062346] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [14996.040317] Key type lgssc unregistered [14996.527891] LNet: 461473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14996.544597] LNetError: 461473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15090.790169] LNet: Removed LNI 192.168.202.137@tcp [15092.106715] Key type .llcrypt unregistered [15092.113218] Key type ._llcrypt unregistered [15112.651671] Key type ._llcrypt registered [15112.653347] Key type .llcrypt registered [15112.743252] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [15114.046503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15114.229917] alg: No test for adler32 (adler32-zlib) [15115.296362] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [15115.486477] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [15117.135154] Key type lgssc registered [15118.244798] Lustre: Echo OBD driver; http://www.lustre.org/ [15122.804741] LNet: Added LNI 192.168.202.137@tcp100 [8/256/0/180] [15123.801570] LNet: Added LNI 192.168.202.137@tcp101 [8/256/0/180] [15124.785889] LNet: Added LNI 192.168.202.137@tcp102 [8/256/0/180] [15125.797750] LNet: Added LNI 192.168.202.137@tcp103 [8/256/0/180] [15126.933412] LNet: Added LNI 192.168.202.137@tcp104 [8/256/0/180] [15127.995931] LNet: Added LNI 192.168.202.137@tcp105 [8/256/0/180] [15128.896452] LNet: Added LNI 192.168.202.137@tcp106 [8/256/0/180] [15129.840955] LNet: Added LNI 192.168.202.137@tcp107 [8/256/0/180] [15130.907266] LNet: Added LNI 192.168.202.137@tcp108 [8/256/0/180] [15131.830268] LNet: Added LNI 192.168.202.137@tcp109 [8/256/0/180] [15132.713500] LNet: Added LNI 192.168.202.137@tcp110 [8/256/0/180] [15133.745840] LNet: Added LNI 192.168.202.137@tcp111 [8/256/0/180] [15134.566792] LNet: Added LNI 192.168.202.137@tcp112 [8/256/0/180] [15135.471669] LNet: Added LNI 192.168.202.137@tcp113 [8/256/0/180] [15136.505315] LNet: Added LNI 192.168.202.137@tcp114 [8/256/0/180] [15137.556962] LNet: Added LNI 192.168.202.137@tcp115 [8/256/0/180] [15138.515411] LNet: Added LNI 192.168.202.137@tcp116 [8/256/0/180] [15139.345677] LNet: Added LNI 192.168.202.137@tcp117 [8/256/0/180] [15140.373932] LNet: Added LNI 192.168.202.137@tcp118 [8/256/0/180] [15141.323821] LNet: Added LNI 192.168.202.137@tcp119 [8/256/0/180] [15142.303943] LNet: Added LNI 192.168.202.137@tcp120 [8/256/0/180] [15143.138509] LNet: Added LNI 192.168.202.137@tcp121 [8/256/0/180] [15144.010630] LNet: Added LNI 192.168.202.137@tcp122 [8/256/0/180] [15144.850950] LNet: Added LNI 192.168.202.137@tcp123 [8/256/0/180] [15145.703438] LNet: Added LNI 192.168.202.137@tcp124 [8/256/0/180] [15146.667722] LNet: Added LNI 192.168.202.137@tcp125 [8/256/0/180] [15147.581644] LNet: Added LNI 192.168.202.137@tcp126 [8/256/0/180] [15148.533876] LNet: Added LNI 192.168.202.137@tcp127 [8/256/0/180] [15149.513213] LNet: Added LNI 192.168.202.137@tcp128 [8/256/0/180] [15150.300495] LNet: Added LNI 192.168.202.137@tcp129 [8/256/0/180] [15151.213330] LNet: Added LNI 192.168.202.137@tcp130 [8/256/0/180] [15152.188494] LNet: Added LNI 192.168.202.137@tcp131 [8/256/0/180] [15153.283372] LNet: Added LNI 192.168.202.137@tcp132 [8/256/0/180] [15154.443912] LNet: Added LNI 192.168.202.137@tcp133 [8/256/0/180] [15155.413825] LNet: Added LNI 192.168.202.137@tcp134 [8/256/0/180] [15156.501958] LNet: Added LNI 192.168.202.137@tcp135 [8/256/0/180] [15163.485491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15163.515059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15164.695013] Lustre: 464501:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.137@tcp: can't add NID 192.168.202.137@tcp130: rc = -75 [15164.802138] LNetError: 461961:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.137 [15164.831969] LNetError: Protocol error connecting to 192.168.202.137@tcp120 at host 192.168.202.137:988: is it running a compatible version of Lustre? [15164.869855] LNetError: Skipped 3 previous similar messages [15165.010785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15165.087128] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15165.202976] Lustre: lustre-MDT0000: new disk, initializing [15165.300330] Lustre: 464558:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.137@tcp: can't add NID 192.168.202.137@tcp130: rc = -75 [15165.308169] Lustre: 464558:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15165.350363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15165.376159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15165.666466] LNet: 5 local NIs in recovery (showing 5): 192.168.202.137@tcp124, 192.168.202.137@tcp126, 192.168.202.137@tcp128, 192.168.202.137@tcp129, 192.168.202.137@tcp120 [15165.669243] LNetError: 464522:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.137 [15165.704541] LNetError: 464522:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [15165.741559] LNetError: Protocol error connecting to 192.168.202.137@tcp at host 192.168.202.137:988: is it running a compatible version of Lustre? [15165.754282] LNetError: Skipped 5 previous similar messages [15170.247956] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [15181.883906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15182.039964] Lustre: 464517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15182.095714] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15182.099113] Lustre: Skipped 1 previous similar message [15182.170903] Lustre: lustre-MDT0001: new disk, initializing [15182.206912] Lustre: 465322:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.137@tcp: can't add NID 192.168.202.137@tcp130: rc = -75 [15182.213409] Lustre: 465322:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15182.311254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15182.340154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15182.356267] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15187.413625] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [15188.483296] Lustre: 465653:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.137@tcp: can't add NID 192.168.202.137@tcp130: rc = -75 [15188.490884] Lustre: 465653:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15194.344782] LNetError: 461963:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.137 [15194.358074] LNetError: 461963:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [15194.371404] LNetError: Protocol error connecting to 192.168.202.137@tcp127 at host 192.168.202.137:988: is it running a compatible version of Lustre? [15194.397872] LNetError: Skipped 1 previous similar message [15197.278715] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15197.413429] LNetError: 464527:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.137 [15197.432462] LNetError: 464527:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [15197.443227] LNetError: Protocol error connecting to 192.168.202.137@tcp125 at host 192.168.202.137:988: is it running a compatible version of Lustre? [15197.458369] LNetError: Skipped 1 previous similar message [15204.248126] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15205.600385] LNetError: 461960:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.137 [15205.617337] LNetError: 461960:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [15205.633612] LNetError: Protocol error connecting to 192.168.202.137@tcp129 at host 192.168.202.137:988: is it running a compatible version of Lustre? [15205.654911] LNetError: Skipped 4 previous similar messages [15211.047161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15211.381559] Lustre: lustre-OST0000: new disk, initializing [15211.385052] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15211.392183] Lustre: 466495:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15211.477606] Lustre: 466500:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.137@tcp: can't add NID 192.168.202.137@tcp130: rc = -75 [15211.482939] Lustre: 466500:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15211.525398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15213.858308] LNetError: 464535:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.137 [15213.866492] LNetError: 464535:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [15213.884667] LNetError: Protocol error connecting to 192.168.202.137@tcp128 at host 192.168.202.137:988: is it running a compatible version of Lustre? [15213.899924] LNetError: Skipped 6 previous similar messages [15217.849067] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [15220.250691] Lustre: 466952:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.137@tcp: can't add NID 192.168.202.137@tcp130: rc = -75 [15220.256806] Lustre: 466952:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15220.311480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15220.320344] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15220.469893] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15227.208534] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15230.250276] LNetError: 461963:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.137 [15230.258750] LNetError: 461963:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [15230.267307] LNetError: Protocol error connecting to 192.168.202.137@tcp129 at host 192.168.202.137:988: is it running a compatible version of Lustre? [15230.279404] LNetError: Skipped 8 previous similar messages [15236.066767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15236.068466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15236.123515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15241.186408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15241.191935] Lustre: Skipped 1 previous similar message [15241.979038] Lustre: server umount lustre-OST0000 complete [15246.306080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15246.324949] Lustre: Skipped 1 previous similar message [15246.337430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15246.347760] Lustre: Skipped 2 previous similar messages [15251.426360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15251.446897] Lustre: Skipped 1 previous similar message [15252.552653] Lustre: server umount lustre-MDT0000 complete [15256.829900] LustreError: 464511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786990198 with bad export cookie 8933401773141075452 [15256.841359] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15256.850570] LustreError: 464511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15257.317293] Lustre: server umount lustre-MDT0001 complete [15263.026987] LNetError: 464527:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.137 [15263.040746] LNetError: 464527:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [15263.048689] LNetError: Protocol error connecting to 192.168.202.137@tcp120 at host 192.168.202.137:988: is it running a compatible version of Lustre? [15263.065701] LNetError: Skipped 11 previous similar messages [15272.810817] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [15276.038256] Key type lgssc unregistered [15276.625871] LNet: 468413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15276.659467] LNetError: 468413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15276.668689] LNet: Removed LNI 192.168.202.137@tcp [15277.727167] LNet: Removed LNI 192.168.202.137@tcp100 [15277.729131] LNetError: 468413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15277.741749] LNetError: 468413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15277.746598] LNet: Removed LNI 192.168.202.137@tcp101 [15278.755158] LNet: Removed LNI 192.168.202.137@tcp102 [15278.761344] LNetError: 468413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15278.775031] LNetError: 468413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15278.783849] LNet: Removed LNI 192.168.202.137@tcp103 [15279.839215] LNet: Removed LNI 192.168.202.137@tcp104 [15280.863668] LNet: Removed LNI 192.168.202.137@tcp105 [15280.867382] LNetError: 468413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15280.882393] LNetError: 468413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15280.903531] LNet: Removed LNI 192.168.202.137@tcp106 [15281.953361] LNet: Removed LNI 192.168.202.137@tcp107 [15282.975479] LNet: Removed LNI 192.168.202.137@tcp108 [15284.000360] LNet: Removed LNI 192.168.202.137@tcp109 [15285.026634] LNet: Removed LNI 192.168.202.137@tcp110 [15285.028878] LNetError: 468413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15285.037579] LNetError: 468413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15285.064750] LNet: Removed LNI 192.168.202.137@tcp111 [15286.113380] LNet: Removed LNI 192.168.202.137@tcp112 [15287.200766] LNet: Removed LNI 192.168.202.137@tcp113 [15288.228096] LNet: Removed LNI 192.168.202.137@tcp114 [15289.247293] LNet: Removed LNI 192.168.202.137@tcp115 [15290.275425] LNet: Removed LNI 192.168.202.137@tcp116 [15291.296067] LNet: Removed LNI 192.168.202.137@tcp117 [15292.319861] LNet: Removed LNI 192.168.202.137@tcp118 [15293.343295] LNet: Removed LNI 192.168.202.137@tcp119 [15293.347860] LNetError: 468413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15293.354312] LNetError: 468413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15293.360988] LNet: Removed LNI 192.168.202.137@tcp120 [15294.367442] LNet: Removed LNI 192.168.202.137@tcp121 [15295.391203] LNet: Removed LNI 192.168.202.137@tcp122 [15296.417387] LNet: Removed LNI 192.168.202.137@tcp123 [15297.439136] LNet: Removed LNI 192.168.202.137@tcp124 [15298.463468] LNet: Removed LNI 192.168.202.137@tcp125 [15299.488263] LNet: Removed LNI 192.168.202.137@tcp126 [15300.511716] LNet: Removed LNI 192.168.202.137@tcp127 [15301.535104] LNet: Removed LNI 192.168.202.137@tcp128 [15302.559205] LNet: Removed LNI 192.168.202.137@tcp129 [15302.561362] LNet: Removed LNI 192.168.202.137@tcp130 [15302.567550] LNet: Removed LNI 192.168.202.137@tcp131 [15302.575741] LNet: Removed LNI 192.168.202.137@tcp132 [15302.582882] LNet: Removed LNI 192.168.202.137@tcp133 [15302.589572] LNet: Removed LNI 192.168.202.137@tcp134 [15302.597667] LNet: Removed LNI 192.168.202.137@tcp135 [15303.487336] Key type .llcrypt unregistered [15303.489443] Key type ._llcrypt unregistered [15328.139393] Key type ._llcrypt registered [15328.142141] Key type .llcrypt registered [15328.222334] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [15347.407958] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [15348.974741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15349.008152] alg: No test for adler32 (adler32-zlib) [15350.085424] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [15350.283819] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [15351.935179] Key type lgssc registered [15352.656424] Lustre: Echo OBD driver; http://www.lustre.org/ [15387.865542] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 14:12:08 (1786990328) [15409.287950] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [15412.525467] Key type lgssc unregistered [15412.898744] LNet: 472507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15412.909733] LNetError: 472507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15412.943971] LNet: Removed LNI 192.168.202.137@tcp [15414.242452] Key type .llcrypt unregistered [15414.249147] Key type ._llcrypt unregistered [15434.028903] Key type ._llcrypt registered [15434.032201] Key type .llcrypt registered [15434.118343] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [15435.347858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15435.419585] alg: No test for adler32 (adler32-zlib) [15436.530494] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [15436.761313] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [15438.423175] Key type lgssc registered [15439.416813] Lustre: Echo OBD driver; http://www.lustre.org/ [15449.772668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15449.811614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15451.231713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15451.309828] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15451.495925] Lustre: lustre-MDT0000: new disk, initializing [15451.605957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15451.630893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15457.781967] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [15475.236098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15475.433196] Lustre: 473721:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15475.634247] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15475.642301] Lustre: Skipped 1 previous similar message [15475.748285] Lustre: lustre-MDT0001: new disk, initializing [15475.902720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15475.968417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15475.989351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15481.607841] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [15492.270835] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15498.647971] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15506.652896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15506.901501] Lustre: lustre-OST0000: new disk, initializing [15506.905256] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15506.913658] Lustre: 475689:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15507.023333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15514.498348] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [15515.258346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15515.275398] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15515.364621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15525.098610] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15529.252959] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing lnet_if_list [15534.814375] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15534.871365] LNet: Added LNI 192.168.202.137@tcp10 [8/256/0/180] [15540.490449] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15540.510993] LNet: Added LNI 192.168.202.137@tcp11 [8/256/0/180] [15545.298797] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15545.341721] LNet: Added LNI 192.168.202.137@tcp12 [8/256/0/180] [15549.867553] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15549.895569] LNet: Added LNI 192.168.202.137@tcp13 [8/256/0/180] [15554.280563] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15554.319516] LNet: Added LNI 192.168.202.137@tcp14 [8/256/0/180] [15559.978682] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15560.017556] LNet: Added LNI 192.168.202.137@tcp15 [8/256/0/180] [15566.298831] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15566.320770] LNet: Added LNI 192.168.202.137@tcp16 [8/256/0/180] [15573.456953] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15573.508036] LNet: Added LNI 192.168.202.137@tcp17 [8/256/0/180] [15579.523319] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15579.549957] LNet: Added LNI 192.168.202.137@tcp18 [8/256/0/180] [15584.676754] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15584.700728] LNet: Added LNI 192.168.202.137@tcp19 [8/256/0/180] [15591.233694] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15591.272040] LNet: Added LNI 192.168.202.137@tcp20 [8/256/0/180] [15597.768543] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15597.801487] LNet: Added LNI 192.168.202.137@tcp21 [8/256/0/180] [15604.411326] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15604.477441] LNet: Added LNI 192.168.202.137@tcp22 [8/256/0/180] [15610.302328] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15610.318607] LNet: Added LNI 192.168.202.137@tcp23 [8/256/0/180] [15616.412347] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15616.463817] LNet: Added LNI 192.168.202.137@tcp24 [8/256/0/180] [15622.868500] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15622.890521] LNet: Added LNI 192.168.202.137@tcp25 [8/256/0/180] [15629.331843] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15629.356898] LNet: Added LNI 192.168.202.137@tcp26 [8/256/0/180] [15636.112746] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15636.144096] LNet: Added LNI 192.168.202.137@tcp27 [8/256/0/180] [15642.769988] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15642.806141] LNet: Added LNI 192.168.202.137@tcp28 [8/256/0/180] [15648.949689] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15648.968040] LNet: Added LNI 192.168.202.137@tcp29 [8/256/0/180] [15655.375375] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15655.407408] LNet: Added LNI 192.168.202.137@tcp30 [8/256/0/180] [15662.896519] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15662.961967] LNet: Added LNI 192.168.202.137@tcp31 [8/256/0/180] [15670.750545] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15670.779642] LNet: Added LNI 192.168.202.137@tcp32 [8/256/0/180] [15678.527733] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15678.558364] LNet: Added LNI 192.168.202.137@tcp33 [8/256/0/180] [15686.448869] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15686.521802] LNet: Added LNI 192.168.202.137@tcp34 [8/256/0/180] [15694.733284] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15694.769486] LNet: Added LNI 192.168.202.137@tcp35 [8/256/0/180] [15701.798063] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15701.828534] LNet: Added LNI 192.168.202.137@tcp36 [8/256/0/180] [15709.049291] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15709.087873] LNet: Added LNI 192.168.202.137@tcp37 [8/256/0/180] [15716.118506] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15716.151934] LNet: Added LNI 192.168.202.137@tcp38 [8/256/0/180] [15724.867225] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15724.913589] LNet: Added LNI 192.168.202.137@tcp39 [8/256/0/180] [15732.279463] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15732.301119] LNet: Added LNI 192.168.202.137@tcp40 [8/256/0/180] [15740.076420] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15740.101373] LNet: Added LNI 192.168.202.137@tcp41 [8/256/0/180] [15748.217758] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15748.237662] LNet: Added LNI 192.168.202.137@tcp42 [8/256/0/180] [15756.973393] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15757.003877] LNet: Added LNI 192.168.202.137@tcp43 [8/256/0/180] [15763.989315] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15764.013424] LNet: Added LNI 192.168.202.137@tcp44 [8/256/0/180] [15772.001894] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15772.039830] LNet: Added LNI 192.168.202.137@tcp45 [8/256/0/180] [15781.209149] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15781.244499] LNet: Added LNI 192.168.202.137@tcp46 [8/256/0/180] [15791.282858] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15791.308994] LNet: Added LNI 192.168.202.137@tcp47 [8/256/0/180] [15799.683613] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15799.719616] LNet: Added LNI 192.168.202.137@tcp48 [8/256/0/180] [15808.887574] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15808.941582] LNet: Added LNI 192.168.202.137@tcp49 [8/256/0/180] [15817.755676] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15817.814285] LNet: Added LNI 192.168.202.137@tcp50 [8/256/0/180] [15838.183825] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15838.185603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15838.202128] Lustre: Skipped 1 previous similar message [15842.550065] Lustre: server umount lustre-MDT0000 complete [15845.987606] LustreError: 473713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786990788 with bad export cookie 5325677544826487959 [15845.991076] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15845.995793] LustreError: 473713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15846.404709] Lustre: server umount lustre-MDT0001 complete [15864.810329] Lustre: 472992:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786990790/real 1786990790] req@00000000079685ae x1873795222776576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786990806 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15864.858095] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15864.876770] Lustre: Skipped 1 previous similar message [15866.415329] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [15866.857533] Lustre: 472991:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786990792/real 1786990792] req@00000000ab3e1971 x1873795222776704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786990808 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15868.904214] Lustre: 472994:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786990795/real 1786990795] req@00000000b4875a08 x1873795222776832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786990811 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15872.033777] Lustre: 472994:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786990798/real 1786990798] req@00000000cc56ee47 x1873795222777088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786990814 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15875.031412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15877.087660] Lustre: 472991:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786990803/real 1786990803] req@0000000002aba3e5 x1873795222777216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786990819 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15891.787136] LustreError: 475683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15891.795482] LustreError: 475683:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15891.818375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15893.857858] LustreError: 475684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15893.862528] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15895.907034] LustreError: 475683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15896.414561] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [15901.165577] LustreError: 475684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15905.809494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15906.246250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15910.314735] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15911.317412] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [15920.347548] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15927.054498] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15953.392521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15953.401751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15953.428467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15953.449969] Lustre: Skipped 2 previous similar messages [15956.460954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15956.462559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15958.836441] Lustre: server umount lustre-OST0000 complete [15966.690863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15966.716751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15966.731823] Lustre: Skipped 2 previous similar messages [15968.994772] Lustre: server umount lustre-MDT0000 complete [15971.813228] LustreError: 499939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15971.822326] LustreError: 499939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15972.894521] LustreError: 473711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786990915 with bad export cookie 5325677544826490465 [15972.902130] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15972.908756] LustreError: 473711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15973.208840] Lustre: server umount lustre-MDT0001 complete [15986.518907] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [15989.714718] Key type lgssc unregistered [15990.280407] LNet: 502741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15990.289988] LNetError: 502741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15990.292738] LNet: Removed LNI 192.168.202.137@tcp [15990.295514] LNet: Removed LNI 192.168.202.137@tcp10 [15990.301436] LNet: Removed LNI 192.168.202.137@tcp11 [15990.304663] LNet: Removed LNI 192.168.202.137@tcp12 [15990.308288] LNet: Removed LNI 192.168.202.137@tcp13 [15990.310551] LNet: Removed LNI 192.168.202.137@tcp14 [15990.315457] LNet: Removed LNI 192.168.202.137@tcp15 [15990.317909] LNet: Removed LNI 192.168.202.137@tcp16 [15990.320141] LNet: Removed LNI 192.168.202.137@tcp17 [15990.321940] LNet: Removed LNI 192.168.202.137@tcp18 [15990.324470] LNet: Removed LNI 192.168.202.137@tcp19 [15990.332422] LNet: Removed LNI 192.168.202.137@tcp20 [15990.334465] LNet: Removed LNI 192.168.202.137@tcp21 [15990.345428] LNet: Removed LNI 192.168.202.137@tcp22 [15990.351015] LNet: Removed LNI 192.168.202.137@tcp23 [15990.355546] LNet: Removed LNI 192.168.202.137@tcp24 [15990.362687] LNet: Removed LNI 192.168.202.137@tcp25 [15990.365478] LNet: Removed LNI 192.168.202.137@tcp26 [15990.368334] LNet: Removed LNI 192.168.202.137@tcp27 [15990.370969] LNet: Removed LNI 192.168.202.137@tcp28 [15990.377277] LNet: Removed LNI 192.168.202.137@tcp29 [15990.380900] LNet: Removed LNI 192.168.202.137@tcp30 [15990.383440] LNet: Removed LNI 192.168.202.137@tcp31 [15990.388311] LNet: Removed LNI 192.168.202.137@tcp32 [15990.394388] LNet: Removed LNI 192.168.202.137@tcp33 [15990.398279] LNet: Removed LNI 192.168.202.137@tcp34 [15990.406297] LNet: Removed LNI 192.168.202.137@tcp35 [15990.409513] LNet: Removed LNI 192.168.202.137@tcp36 [15990.412761] LNet: Removed LNI 192.168.202.137@tcp37 [15990.417660] LNet: Removed LNI 192.168.202.137@tcp38 [15990.429589] LNet: Removed LNI 192.168.202.137@tcp39 [15990.436976] LNet: Removed LNI 192.168.202.137@tcp40 [15990.444230] LNet: Removed LNI 192.168.202.137@tcp41 [15990.447321] LNet: Removed LNI 192.168.202.137@tcp42 [15990.454285] LNet: Removed LNI 192.168.202.137@tcp43 [15990.458361] LNet: Removed LNI 192.168.202.137@tcp44 [15990.463276] LNet: Removed LNI 192.168.202.137@tcp45 [15990.470696] LNet: Removed LNI 192.168.202.137@tcp46 [15990.473762] LNet: Removed LNI 192.168.202.137@tcp47 [15990.478571] LNet: Removed LNI 192.168.202.137@tcp48 [15990.482283] LNet: Removed LNI 192.168.202.137@tcp49 [15990.501731] LNet: Removed LNI 192.168.202.137@tcp50 [15991.409813] Key type .llcrypt unregistered [15991.411693] Key type ._llcrypt unregistered [16013.039896] Key type ._llcrypt registered [16013.042588] Key type .llcrypt registered [16013.150659] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [16028.783580] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [16030.240096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16030.283893] alg: No test for adler32 (adler32-zlib) [16031.370525] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [16031.608531] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [16033.255457] Key type lgssc registered [16034.255206] Lustre: Echo OBD driver; http://www.lustre.org/ [16066.069373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16066.098769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16067.328446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16067.353628] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16067.424410] Lustre: lustre-MDT0000: new disk, initializing [16067.494112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16067.510529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16071.321308] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16080.752615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16080.867548] Lustre: 505943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16080.912848] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16080.919373] Lustre: Skipped 1 previous similar message [16081.001709] Lustre: lustre-MDT0001: new disk, initializing [16081.094799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16081.124261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16081.135585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16085.499933] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16094.855419] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16101.200693] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16109.081888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16109.340279] Lustre: *** cfs_fail_loc=90f, val=0*** [16109.440147] Lustre: lustre-OST0000: new disk, initializing [16109.443367] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16109.464962] Lustre: 507956:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16109.550244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16114.765451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16114.774275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16114.900049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16116.412360] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16126.844761] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16135.651084] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16135.655380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16135.668172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16135.672147] Lustre: Skipped 1 previous similar message [16140.306634] Lustre: server umount lustre-OST0000 complete [16145.892242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16145.897485] Lustre: Skipped 1 previous similar message [16145.904390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16145.911990] Lustre: Skipped 1 previous similar message [16151.009467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16151.024654] Lustre: Skipped 1 previous similar message [16155.106951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16159.199303] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16159.407848] Lustre: server umount lustre-MDT0000 complete [16163.112583] LustreError: 505934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786991105 with bad export cookie 12340058147458104693 [16163.123735] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16163.129265] LustreError: 505934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16163.586386] Lustre: server umount lustre-MDT0001 complete [16175.502926] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [16178.290594] Key type lgssc unregistered [16178.522656] LNet: 509868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16178.531443] LNetError: 509868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16178.556128] LNet: Removed LNI 192.168.202.137@tcp [16179.534237] Key type .llcrypt unregistered [16179.536555] Key type ._llcrypt unregistered [16199.920857] Key type ._llcrypt registered [16199.923190] Key type .llcrypt registered [16200.084221] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [16202.031279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16202.148543] alg: No test for adler32 (adler32-zlib) [16203.145456] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [16203.412857] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [16205.071206] Key type lgssc registered [16206.149318] Lustre: Echo OBD driver; http://www.lustre.org/ [16212.925521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16212.937758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16214.208124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16217.935616] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16224.737273] LustreError: 511138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16224.749561] LustreError: 511138:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16226.473763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16226.861442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16231.834969] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16241.080888] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16248.765982] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16255.478765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16255.923616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16262.807261] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16270.927972] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16281.966390] loop0: detected capacity change from 0 to 204800000 [16282.184469] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16282.193197] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16282.204631] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16288.595691] loop0: detected capacity change from 0 to 204800000 [16288.769576] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16288.787730] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16288.803833] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16293.807393] loop0: detected capacity change from 0 to 204800000 [16299.429762] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16299.571636] Lustre: fs15246-OST0000: new disk, initializing [16299.576219] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16299.583407] Lustre: 514733:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16299.633169] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16305.839821] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16316.913846] loop1: detected capacity change from 0 to 204800000 [16323.009292] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16323.144509] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16323.195604] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16323.297363] Lustre: fs15246-MDT0000: new disk, initializing [16323.471442] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16323.495265] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16323.621664] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16324.705596] Lustre: 515857:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000092d44555 x1873796026908416/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786991282 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16327.655961] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16329.832565] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16329.842697] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16329.878134] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16350.177525] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16350.186686] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16355.305615] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16355.856036] Lustre: server umount fs15246-MDT0000 complete [16360.423413] LustreError: 511132:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16365.049739] Lustre: server umount fs15246-OST0000 complete [16378.848105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16378.862165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16378.882508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16380.923212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16380.931420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16383.417242] Lustre: server umount lustre-OST0000 complete [16390.625398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16390.632329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16390.643193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16390.649083] Lustre: Skipped 1 previous similar message [16392.981299] Lustre: server umount lustre-MDT0000 complete [16396.257256] LustreError: 511138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16396.272924] LustreError: 511138:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16396.531466] LustreError: 517999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786991338 with bad export cookie 1785578672575451615 [16396.536734] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16396.545438] LustreError: 517999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16396.802263] Lustre: server umount lustre-MDT0001 complete [16407.938861] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [16410.842723] Key type lgssc unregistered [16411.223526] LNet: 518957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16411.251802] LNetError: 518957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16412.362178] LNet: Removed LNI 192.168.202.137@tcp [16413.696722] Key type .llcrypt unregistered [16413.701388] Key type ._llcrypt unregistered [16437.839476] Key type ._llcrypt registered [16437.842353] Key type .llcrypt registered [16437.922655] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [16439.171440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16439.393968] alg: No test for adler32 (adler32-zlib) [16440.558465] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [16440.931155] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [16442.647195] Key type lgssc registered [16443.810594] Lustre: Echo OBD driver; http://www.lustre.org/ [16485.888862] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [16497.207818] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [16540.663703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16540.693417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16541.987465] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16542.018164] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16542.102481] Lustre: lustre-MDT0000: new disk, initializing [16542.199747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16542.212639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16546.127219] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16556.642465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16556.703325] Lustre: 523429:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16556.750704] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16556.753853] Lustre: Skipped 1 previous similar message [16556.822976] Lustre: lustre-MDT0001: new disk, initializing [16556.896532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16556.919297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16556.923667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16561.083508] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16571.047536] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16578.206839] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16585.885800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16586.113110] Lustre: lustre-OST0000: new disk, initializing [16586.116178] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16586.120958] Lustre: 525397:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16586.195189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16591.452175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16591.461236] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16591.606907] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16592.692924] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16603.233584] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16607.619670] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16607.795582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16613.354501] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16613.630983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16617.442217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16617.454802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16617.490553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16617.497764] Lustre: Skipped 1 previous similar message [16622.719797] Lustre: server umount lustre-OST0000 complete [16627.697228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16627.700866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16627.733235] Lustre: Skipped 2 previous similar messages [16627.760670] Lustre: Skipped 1 previous similar message [16632.807811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16632.811932] Lustre: Skipped 1 previous similar message [16633.829937] Lustre: server umount lustre-MDT0000 complete [16637.853770] LustreError: 523419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786991580 with bad export cookie 9805102142988695544 [16637.858789] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16637.865312] LustreError: 523419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16638.153694] Lustre: server umount lustre-MDT0001 complete [16663.349687] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [16675.071842] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [16710.053046] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16390 sec ========= 14:34:10 (1786991650) [16712.550116] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:34:12 (1786991652) === [16718.522395] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:34:17 (1786991657) === [16768.260875] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [16770.617675] Key type lgssc unregistered [16770.879963] LNet: 532047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16770.884912] LNetError: 532047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16770.897657] LNet: Removed LNI 192.168.202.137@tcp [16771.614361] Key type .llcrypt unregistered [16771.616230] Key type ._llcrypt unregistered