[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 671881538 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.007531] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010120] LSM: Security Framework initializing [ 0.012045] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.015019] *** VALIDATE selinux *** [ 0.024100] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029525] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030141] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032093] *** VALIDATE tmpfs *** [ 0.033438] *** VALIDATE proc *** [ 0.035094] *** VALIDATE cgroup *** [ 0.036007] *** VALIDATE cgroup2 *** [ 0.038240] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039171] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041030] Spectre V2 : User space: Vulnerable [ 0.042019] Speculative Store Bypass: Vulnerable [ 0.045151] debug: unmapping init [mem 0xffffffff85459000-0xffffffff85460fff] [ 0.048000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048605] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049025] ... version: 2 [ 0.050014] ... bit width: 48 [ 0.051024] ... generic registers: 4 [ 0.052017] ... value mask: 0000ffffffffffff [ 0.053027] ... max period: 00007fffffffffff [ 0.054019] ... fixed-purpose events: 3 [ 0.055018] ... event mask: 000000070000000f [ 0.056442] rcu: Hierarchical SRCU implementation. [ 0.058673] smp: Bringing up secondary CPUs ... [ 0.059627] x86: Booting SMP configuration: [ 0.060027] .... node #0, CPUs: #1 #2 #3 [ 0.073137] smp: Brought up 1 node, 4 CPUs [ 0.075016] smpboot: Max logical packages: 1 [ 0.076014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.112026] node 0 deferred pages initialised in 34ms [ 0.114180] devtmpfs: initialized [ 0.117387] x86/mm: Memory block size: 128MB [ 0.123304] gcov: version magic: 0x41383552 [ 0.128000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.133087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.138278] pinctrl core: initialized pinctrl subsystem [ 0.142422] [ 0.143024] ************************************************************* [ 0.148013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152010] ** ** [ 0.156012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160013] ** ** [ 0.164012] ** This means that this kernel is built to expose internal ** [ 0.167011] ** IOMMU data structures, which may compromise security on ** [ 0.171014] ** your system. ** [ 0.175026] ** ** [ 0.180036] ** If you see this message and you are not debugging the ** [ 0.185013] ** kernel, report this immediately to your vendor! ** [ 0.187013] ** ** [ 0.191019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195106] ************************************************************* [ 0.198139] NET: Registered protocol family 16 [ 0.200873] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.206075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.211063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.218092] cpuidle: using governor menu [ 0.221307] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.223594] PCI: Using configuration type 1 for base access [ 0.228136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.238044] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.244034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.253035] cryptd: max_cpu_qlen set to 1000 [ 0.258189] ACPI: Added _OSI(Module Device) [ 0.261016] ACPI: Added _OSI(Processor Device) [ 0.265014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.270014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.280299] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.290414] ACPI: Interpreter enabled [ 0.292064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.297052] ACPI: Using IOAPIC for interrupt routing [ 0.300176] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.305510] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.321176] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.324035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.330021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.335072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.343414] acpiphp: Slot [2] registered [ 0.345225] acpiphp: Slot [5] registered [ 0.347121] acpiphp: Slot [6] registered [ 0.350219] acpiphp: Slot [7] registered [ 0.353128] acpiphp: Slot [8] registered [ 0.355143] acpiphp: Slot [9] registered [ 0.357267] acpiphp: Slot [10] registered [ 0.360194] acpiphp: Slot [3] registered [ 0.362117] acpiphp: Slot [4] registered [ 0.364134] acpiphp: Slot [11] registered [ 0.366246] acpiphp: Slot [12] registered [ 0.368124] acpiphp: Slot [13] registered [ 0.371143] acpiphp: Slot [14] registered [ 0.373196] acpiphp: Slot [15] registered [ 0.376163] acpiphp: Slot [16] registered [ 0.378189] acpiphp: Slot [17] registered [ 0.380240] acpiphp: Slot [18] registered [ 0.383136] acpiphp: Slot [19] registered [ 0.385116] acpiphp: Slot [20] registered [ 0.387131] acpiphp: Slot [21] registered [ 0.390136] acpiphp: Slot [22] registered [ 0.391114] acpiphp: Slot [23] registered [ 0.393202] acpiphp: Slot [24] registered [ 0.396124] acpiphp: Slot [25] registered [ 0.399135] acpiphp: Slot [26] registered [ 0.401106] acpiphp: Slot [27] registered [ 0.404207] acpiphp: Slot [28] registered [ 0.406130] acpiphp: Slot [29] registered [ 0.408209] acpiphp: Slot [30] registered [ 0.411224] acpiphp: Slot [31] registered [ 0.413081] PCI host bridge to bus 0000:00 [ 0.416029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.418042] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.422022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.426053] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.430026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.435037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.437223] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.441685] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.446412] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.456000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.464057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.468017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.471015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.475020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.479358] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.482877] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.487042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.491820] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.497011] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.509978] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.516962] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.524115] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.538014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.550013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.590014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.606878] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.619013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.635014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.670017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.685806] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.701013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.711021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.737015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.753981] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.767020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.777017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.809016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.830513] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.847012] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.864014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.906013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.925000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.937000] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.952019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.989015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.003599] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.006451] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.009440] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.012500] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.016201] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.022531] iommu: Default domain type: Passthrough [ 1.023000] SCSI subsystem initialized [ 1.023093] ACPI: bus type USB registered [ 1.026117] usbcore: registered new interface driver usbfs [ 1.029133] usbcore: registered new interface driver hub [ 1.031102] usbcore: registered new device driver usb [ 1.033149] pps_core: LinuxPPS API ver. 1 registered [ 1.036013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.044052] PTP clock support registered [ 1.046126] EDAC MC: Ver: 3.0.0 [ 1.047139] PCI: Using ACPI for IRQ routing [ 1.049000] NetLabel: Initializing [ 1.052010] NetLabel: domain hash size = 128 [ 1.054010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.056216] NetLabel: unlabeled traffic allowed by default [ 1.062129] vgaarb: loaded [ 1.065207] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.067012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.074898] clocksource: Switched to clocksource kvm-clock [ 1.197713] VFS: Disk quotas dquot_6.6.0 [ 1.199441] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.202080] *** VALIDATE ramfs *** [ 1.203321] *** VALIDATE hugetlbfs *** [ 1.205171] pnp: PnP ACPI init [ 1.207450] pnp: PnP ACPI: found 6 devices [ 1.229291] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.233994] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.237271] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.239583] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.242636] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.245538] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.249851] NET: Registered protocol family 2 [ 1.252517] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.257635] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.262708] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.270046] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.273698] TCP: Hash tables configured (established 65536 bind 65536) [ 1.277120] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.280498] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.283391] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.286636] NET: Registered protocol family 1 [ 1.289480] RPC: Registered named UNIX socket transport module. [ 1.291975] RPC: Registered udp transport module. [ 1.293516] RPC: Registered tcp transport module. [ 1.295399] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.298163] NET: Registered protocol family 44 [ 1.300536] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.302773] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.305046] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.307549] PCI: CLS 0 bytes, default 64 [ 1.310064] Unpacking initramfs... [ 2.947670] debug: unmapping init [mem 0xffffa04cbcc54000-0xffffa04cbffbffff] [ 2.956452] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.959778] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.967229] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.566721] Initialise system trusted keyrings [ 3.569582] Key type blacklist registered [ 3.572307] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.583577] zbud: loaded [ 3.590101] *** VALIDATE nfs *** [ 3.591847] *** VALIDATE nfs4 *** [ 3.594091] pstore: using deflate compression [ 3.598727] Platform Keyring initialized [ 3.717619] NET: Registered protocol family 38 [ 3.719437] Key type asymmetric registered [ 3.720933] Asymmetric key parser 'x509' registered [ 3.726249] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.733410] io scheduler mq-deadline registered [ 3.735917] io scheduler kyber registered [ 3.738484] io scheduler bfq registered [ 3.741313] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.746665] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.754142] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.759318] ACPI: Power Button [PWRF] [ 3.766307] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.773667] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.819586] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.846160] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.886996] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.922111] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.958328] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.966317] Non-volatile memory driver v1.3 [ 3.967979] Linux agpgart interface v0.103 [ 4.013719] virtio_blk virtio1: [vda] 149944 512-byte logical blocks (76.8 MB/73.2 MiB) [ 4.017130] vda: detected capacity change from 0 to 76771328 [ 4.043389] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.047635] vdb: detected capacity change from 0 to 1073741824 [ 4.073452] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.079214] vdc: detected capacity change from 0 to 2621440000 [ 4.108828] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.116230] vdd: detected capacity change from 0 to 2621440000 [ 4.147481] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.153635] vde: detected capacity change from 0 to 4294967296 [ 4.175436] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.181327] vdf: detected capacity change from 0 to 4294967296 [ 4.196879] libphy: Fixed MDIO Bus: probed [ 4.203552] usbcore: registered new interface driver usbserial_generic [ 4.208601] usbserial: USB Serial support registered for generic [ 4.213511] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.220149] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.222967] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.226836] mousedev: PS/2 mouse device common for all mice [ 4.230707] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.234358] rtc_cmos 00:05: RTC can wake from S4 [ 4.242499] rtc_cmos 00:05: registered as rtc0 [ 4.244379] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.250525] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.255927] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.260049] intel_pstate: CPU model not supported [ 4.274477] hid: raw HID events driver (C) Jiri Kosina [ 4.276233] usbcore: registered new interface driver usbhid [ 4.277496] usbhid: USB HID core driver [ 4.278771] drop_monitor: Initializing network drop monitor service [ 4.280631] Initializing XFRM netlink socket [ 4.283037] NET: Registered protocol family 10 [ 4.287239] Segment Routing with IPv6 [ 4.289524] NET: Registered protocol family 17 [ 4.292562] mpls_gso: MPLS GSO support [ 4.298099] RAS: Correctable Errors collector initialized. [ 4.301191] AVX version of gcm_enc/dec engaged. [ 4.303792] AES CTR mode by8 optimization enabled [ 4.391443] sched_clock: Marking stable (4391424284, 0)->(5620047324, -1228623040) [ 4.397215] registered taskstats version 1 [ 4.400366] Loading compiled-in X.509 certificates [ 4.403115] zswap: loaded using pool lzo/zbud [ 4.434600] Key type big_key registered [ 4.452524] Key type encrypted registered [ 4.455499] ima: No TPM chip found, activating TPM-bypass! [ 4.458851] ima: Allocated hash algorithm: sha1 [ 4.461898] ima: No architecture policies found [ 4.464728] evm: Initialising EVM extended attributes: [ 4.467608] evm: security.selinux [ 4.469659] evm: security.ima [ 4.471529] evm: security.capability [ 4.473975] evm: HMAC attrs: 0x1 [ 4.477596] rtc_cmos 00:05: setting system clock to 2026-09-16 22:01:01 UTC (1789596061) [ 4.487242] debug: unmapping init [mem 0xffffffff86403000-0xffffffff865fffff] [ 4.492502] debug: unmapping init [mem 0xffffffff85182000-0xffffffff85458fff] [ 4.502219] Write protecting the kernel read-only data: 28672k [ 4.508691] debug: unmapping init [mem 0xffffffff83803000-0xffffffff839fffff] [ 4.513474] debug: unmapping init [mem 0xffffffff84114000-0xffffffff841fffff] [ 4.552977] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.567637] systemd[1]: Detected virtualization kvm. [ 4.570728] systemd[1]: Detected architecture x86-64. [ 4.573803] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.604143] systemd[1]: No hostname configured. [ 4.609027] systemd[1]: Set hostname to . [ 4.612261] random: systemd: uninitialized urandom read (16 bytes read) [ 4.616116] systemd[1]: Initializing machine ID from random generator. [ 4.692819] random: ln: uninitialized urandom read (6 bytes read) [ 4.827582] random: systemd: uninitialized urandom read (16 bytes read) [ 4.833223] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.847029] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.861420] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.835432] device-mapper: uevent: version 1.0.3 [ 5.838499] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.988591] virtio_net virtio0 ens2: renamed from eth0 [ 7.104488] random: fast init done [ 7.171964] scsi host0: ata_piix [ 7.182571] scsi host1: ata_piix [ 7.188259] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.196702] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.981753] random: crng init done [ 11.983526] random: 7 urandom warning(s) missed due to ratelimiting [ 12.714285] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 13.612416] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.420971] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.829978] SELinux: Disabled at runtime. [ 15.891985] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.905897] systemd[1]: Detected virtualization kvm. [ 15.908107] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.689201] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.692881] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.701483] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.706114] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.714178] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.731400] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.736209] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ 16.901469] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 17.767554] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.457073] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.750768] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.916491] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.105613] EDAC sbridge: Ver: 1.1.2 [ 22.344674] Key type dns_resolver registered [ 22.694578] NFS: Registering the id_resolver key type [ 22.696809] Key type id_resolver registered [ 22.699247] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg659-server login: [ 56.795894] hrtimer: interrupt took 4109831 ns [ 85.646939] libcfs: loading out-of-tree module taints kernel. [ 85.778874] Key type ._llcrypt registered [ 85.789938] Key type .llcrypt registered [ 85.985706] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 107.216933] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 108.940929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 108.957588] alg: No test for adler32 (adler32-zlib) [ 110.379062] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 111.036375] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 112.720172] Key type lgssc registered [ 114.711035] Lustre: Echo OBD driver; http://www.lustre.org/ [ 128.706748] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 169.300279] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 182.020590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 182.056736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 183.332886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 183.358452] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 183.481941] Lustre: lustre-MDT0000: new disk, initializing [ 183.573698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 183.596598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 187.569494] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.154088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 200.250268] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 200.274574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 200.279724] Lustre: Skipped 1 previous similar message [ 200.376311] Lustre: lustre-MDT0001: new disk, initializing [ 200.440517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 200.471823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 200.491947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 205.178052] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.911549] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 219.243454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.472587] Lustre: lustre-OST0000: new disk, initializing [ 219.476412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 219.483921] Lustre: 8454:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 219.534484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 224.785136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 224.806593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 224.887188] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 225.556814] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.455257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 239.560443] Lustre: lustre-OST0001: new disk, initializing [ 239.563552] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 239.568630] Lustre: 9528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 239.626143] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 246.444523] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.346125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 247.354309] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 247.442509] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 259.501107] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 266.716130] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 274.519227] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing check_logdir /tmp/testlogs/ [ 279.580845] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing yml_node [ 284.915272] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 287.667410] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 290.416672] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 292.092885] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 16 18:05:48 EDT 2026 [ 311.113617] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 312.578970] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 328.676382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 328.686199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 328.698842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.258895] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.262132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.274217] Lustre: Skipped 2 previous similar messages [ 332.289220] Lustre: Skipped 3 previous similar messages [ 334.426131] Lustre: server umount lustre-MDT0000 complete [ 337.970795] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789596394 with bad export cookie 18330688672118086584 [ 337.981295] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 337.991702] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 342.497453] LustreError: 6522:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 342.499546] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.535356] LustreError: 6522:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 342.566097] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 342.573543] Lustre: Skipped 1 previous similar message [ 344.441849] Lustre: server umount lustre-MDT0001 complete [ 348.822676] Lustre: server umount lustre-OST0000 complete [ 352.612496] Lustre: server umount lustre-OST0001 complete [ 359.841801] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 370.496556] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 411.394859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.639673] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 411.696876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 411.797227] Lustre: lustre-MDT0000: new disk, initializing [ 411.938952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 411.961875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 417.451942] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 429.135430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 429.257978] Lustre: 15081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 429.300638] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 429.304399] Lustre: Skipped 1 previous similar message [ 429.369236] Lustre: lustre-MDT0001: new disk, initializing [ 429.486152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 429.566373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 429.574192] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 434.972222] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 445.601757] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 452.583049] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 460.061144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 460.292411] Lustre: lustre-OST0000: new disk, initializing [ 460.296951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 460.304246] Lustre: 17051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 460.403319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 462.133567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 462.140884] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 462.281725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 467.178461] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 477.857901] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 482.848279] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 483.146131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 488.839806] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 489.127679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 493.025657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 493.034304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.037633] LustreError: Skipped 1 previous similar message [ 493.038636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 493.054793] Lustre: Skipped 2 previous similar messages [ 497.697353] Lustre: server umount lustre-OST0000 complete [ 503.268607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.286214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 503.295228] Lustre: Skipped 2 previous similar messages [ 508.388419] LustreError: 15087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.420240] LustreError: 15087:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 508.495592] Lustre: server umount lustre-MDT0000 complete [ 512.631370] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789596569 with bad export cookie 18330688672118088117 [ 512.641784] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 513.177508] Lustre: server umount lustre-MDT0001 complete [ 517.810597] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 18:09:34 (1789596574) [ 520.850974] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 523.043282] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 18:09:39 (1789596579) [ 530.864557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.445573] LustreError: 19462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.578955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 537.065402] LustreError: 19463:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 537.203280] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 541.157805] LustreError: 19462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.973960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 552.461728] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 560.455394] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 566.619408] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 574.511362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 582.131921] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 591.395392] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 606.689566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 606.696536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 606.709228] Lustre: Skipped 1 previous similar message [ 606.717940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 609.067534] Lustre: server umount lustre-OST0000 complete [ 613.857873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 613.866339] LustreError: Skipped 1 previous similar message [ 619.797678] Lustre: server umount lustre-MDT0000 complete [ 623.078804] LustreError: 19463:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 623.099511] LustreError: 19463:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 623.840649] LustreError: 19442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789596680 with bad export cookie 18330688672118088992 [ 623.849887] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 623.859334] LustreError: 19442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 624.197258] Lustre: server umount lustre-MDT0001 complete [ 638.054690] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 641.361646] Key type lgssc unregistered [ 641.709057] LNet: 22995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.715793] LNetError: 22995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 641.732938] LNet: Removed LNI 192.168.206.159@tcp [ 642.943162] Key type .llcrypt unregistered [ 642.944832] Key type ._llcrypt unregistered [ 669.227851] Key type ._llcrypt registered [ 669.232425] Key type .llcrypt registered [ 669.325050] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 670.630458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 670.772925] alg: No test for adler32 (adler32-zlib) [ 672.035868] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 672.317426] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 674.072175] Key type lgssc registered [ 675.430424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 683.749754] loop: module loaded [ 683.830298] loop0: detected capacity change from 0 to 204800000 [ 684.113851] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 684.125170] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 684.465438] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 690.745889] loop0: detected capacity change from 0 to 204800000 [ 690.948220] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 690.958216] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 690.976343] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 696.976938] loop0: detected capacity change from 0 to 204800000 [ 703.524188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 703.560601] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 704.833800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 704.888862] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 705.000557] Lustre: lustre-MDT0000: new disk, initializing [ 705.112928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 705.152598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 710.959289] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 721.216864] loop1: detected capacity change from 0 to 204800000 [ 727.877531] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 728.119639] Lustre: lustre-OST07c6: new disk, initializing [ 728.131334] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 728.142597] Lustre: Skipped 1 previous similar message [ 728.163799] Lustre: 26180:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 728.243372] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 734.858142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 734.869887] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 734.913187] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 735.749345] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 741.908791] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 741.924636] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 741.930543] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 741.939090] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 741.951632] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 741.961765] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 762.813707] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 762.823264] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 762.836653] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 762.848399] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 762.857524] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 762.872367] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 768.170865] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 768.177930] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 768.182795] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 768.189086] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.195318] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 768.200652] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.212553] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 768.222788] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.227990] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 768.238329] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.244130] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 768.248211] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.492608] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 771.507492] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 771.513487] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 771.519440] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.530628] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 771.540738] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.554595] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 771.565180] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.574826] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 771.587268] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.599949] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 771.611653] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.933205] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 776.949181] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 776.959479] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 776.968206] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.972055] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 776.978838] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.989077] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 777.004185] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.015176] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 777.022748] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.029584] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 777.044035] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.239517] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 785.250922] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 785.273273] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 785.281214] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 785.288642] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 785.300826] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 785.315087] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 785.323679] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 785.330048] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 785.335604] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 785.342808] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 785.348791] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 842.209665] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 842.219531] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 842.232049] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 843.885755] Lustre: server umount lustre-OST07c6 complete [ 853.888238] Lustre: server umount lustre-MDT0000 complete [ 871.424860] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 874.296241] Key type lgssc unregistered [ 874.538636] LNet: 28737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.546152] LNetError: 28737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.577953] LNet: Removed LNI 192.168.206.159@tcp [ 875.403081] Key type .llcrypt unregistered [ 875.405530] Key type ._llcrypt unregistered [ 894.147826] Key type ._llcrypt registered [ 894.152279] Key type .llcrypt registered [ 894.245644] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 895.284332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 895.551144] alg: No test for adler32 (adler32-zlib) [ 896.658610] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 896.983254] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 898.698427] Key type lgssc registered [ 899.865351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 907.762744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 907.794723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 909.330287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 914.185948] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 919.524608] LustreError: 30005:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 919.542471] LustreError: 30005:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 922.735628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 923.123312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 928.020799] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 936.027267] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 942.734311] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 950.290882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 950.838891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 954.951333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 957.756954] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 966.815947] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 985.571865] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 985.579880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 985.590745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 989.153926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 989.156257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 989.178096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.275653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 994.291599] Lustre: Skipped 1 previous similar message [ 998.368305] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 998.530555] Lustre: server umount lustre-OST0000 complete [ 1004.523217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1004.539450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1004.548532] Lustre: Skipped 1 previous similar message [ 1004.563468] Lustre: Skipped 2 previous similar messages [ 1009.101381] Lustre: server umount lustre-MDT0000 complete [ 1009.637801] LustreError: 30005:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1009.662025] LustreError: 30005:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1013.223333] LustreError: 31849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789597070 with bad export cookie 15169705561743109100 [ 1013.225037] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1013.244139] LustreError: 31849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1013.606577] Lustre: server umount lustre-MDT0001 complete [ 1025.122464] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 1028.204943] Key type lgssc unregistered [ 1028.571275] LNet: 33740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.580950] LNetError: 33740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1028.594537] LNet: Removed LNI 192.168.206.159@tcp [ 1029.486334] Key type .llcrypt unregistered [ 1029.489282] Key type ._llcrypt unregistered [ 1050.784960] Key type ._llcrypt registered [ 1050.787444] Key type .llcrypt registered [ 1050.952990] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 1052.152997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1052.248548] alg: No test for adler32 (adler32-zlib) [ 1053.247180] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 1053.463604] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 1055.176190] Key type lgssc registered [ 1056.655794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1065.040286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1065.073985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1066.623501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1071.528451] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1076.196459] LustreError: 35010:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1076.210813] LustreError: 35010:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1080.433398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1080.795186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1086.074370] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1093.343691] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1099.319718] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1107.118345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1107.549508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1111.680177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1114.090619] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1121.388093] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1134.566723] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.616398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1139.692578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1139.704018] Lustre: Skipped 2 previous similar messages [ 1140.453340] Lustre: server umount lustre-OST0000 complete [ 1144.802792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.828824] Lustre: Skipped 1 previous similar message [ 1144.840286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1144.845571] Lustre: Skipped 1 previous similar message [ 1149.931880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1149.945092] Lustre: Skipped 1 previous similar message [ 1150.534075] Lustre: server umount lustre-MDT0000 complete [ 1154.503306] LustreError: 34990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789597211 with bad export cookie 7063157449645496040 [ 1154.509493] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1154.514554] LustreError: 34990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1154.897896] Lustre: server umount lustre-MDT0001 complete [ 1165.863243] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 1168.568703] Key type lgssc unregistered [ 1168.818977] LNet: 38567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1168.827053] LNetError: 38567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1168.838244] LNet: Removed LNI 192.168.206.159@tcp [ 1169.613455] Key type .llcrypt unregistered [ 1169.616412] Key type ._llcrypt unregistered [ 1196.573670] Key type ._llcrypt registered [ 1196.575871] Key type .llcrypt registered [ 1196.667722] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 1197.951176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1198.067641] alg: No test for adler32 (adler32-zlib) [ 1199.282358] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 1199.554500] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 1201.264306] Key type lgssc registered [ 1202.404089] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1210.476258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1210.496650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1212.195144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1216.606193] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1222.626883] LustreError: 39982:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1222.641603] LustreError: 39982:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1224.837238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1225.184292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1229.778572] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1237.630826] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1243.039202] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1250.177086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1250.556641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1256.836498] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1257.714366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1263.773138] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1270.498613] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.59@tcp (stopping) [ 1271.267846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1271.277297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1271.292694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1271.548203] Lustre: server umount lustre-MDT0000 complete [ 1273.320900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.328788] LustreError: 40708:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.353193] LustreError: 40708:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1275.031845] LustreError: 40715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789597332 with bad export cookie 15949545703929093677 [ 1275.034248] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1275.043375] LustreError: 40715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1275.602825] LustreError: 39978:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1275.628900] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.59@tcp (stopping) [ 1278.433306] LustreError: 39976:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.437858] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1278.472077] Lustre: Skipped 1 previous similar message [ 1278.483126] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1281.465872] Lustre: server umount lustre-MDT0001 complete [ 1301.737476] Lustre: server umount lustre-OST0000 complete [ 1318.498898] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 1321.491884] Key type lgssc unregistered [ 1321.823858] LNet: 43837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1321.870754] LNetError: 43837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1321.902742] LNet: Removed LNI 192.168.206.159@tcp [ 1322.793724] Key type .llcrypt unregistered [ 1322.796847] Key type ._llcrypt unregistered [ 1348.716365] Key type ._llcrypt registered [ 1348.727362] Key type .llcrypt registered [ 1348.850161] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 1349.738509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1349.854698] alg: No test for adler32 (adler32-zlib) [ 1350.946462] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 1351.202090] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 1352.920333] Key type lgssc registered [ 1353.914208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1361.866331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1361.894648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1363.434852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1367.899769] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1373.671585] LustreError: 45250:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1373.700489] LustreError: 45250:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1376.009543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1376.530364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1381.222586] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1389.182202] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1395.296280] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1402.222116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1402.295866] LustreError: 47072:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1402.616538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1408.728545] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1409.013302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1415.955962] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1423.085588] Lustre: lustre-OST0000: Not available for connect from 192.168.206.59@tcp (stopping) [ 1423.330485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1423.334788] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1424.356640] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1424.358311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1424.359358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1424.382077] Lustre: Skipped 2 previous similar messages [ 1427.782169] Lustre: server umount lustre-OST0000 complete [ 1448.418107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1448.425704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1448.442908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1452.216753] Lustre: server umount lustre-MDT0000 complete [ 1454.565697] LustreError: 45250:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1454.597095] LustreError: 45250:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1455.970688] LustreError: 45232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789597512 with bad export cookie 280353712405942821 [ 1455.971565] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1455.989187] LustreError: 45232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1462.409990] Lustre: server umount lustre-MDT0001 complete [ 1472.653630] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 1475.313609] Key type lgssc unregistered [ 1475.627693] LNet: 48956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1475.639159] LNetError: 48956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1475.655529] LNet: Removed LNI 192.168.206.159@tcp [ 1476.369258] Key type .llcrypt unregistered [ 1476.372986] Key type ._llcrypt unregistered [ 1501.871631] Key type ._llcrypt registered [ 1501.873989] Key type .llcrypt registered [ 1501.968787] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 1502.982936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1503.126033] alg: No test for adler32 (adler32-zlib) [ 1504.269095] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 1504.500383] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 1506.208758] Key type lgssc registered [ 1507.204266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1514.102343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1514.122166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1515.513209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1519.610846] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1525.731186] LustreError: 50369:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1525.743934] LustreError: 50369:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1527.788483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1528.237628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1532.578794] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1539.265404] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1545.009141] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1551.231727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1551.531790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1556.571610] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1560.756989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1563.743981] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1568.648208] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1571.903339] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1584.371790] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-1618h27m20s [ 1585.862797] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1589.730566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1589.738670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1589.753892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1590.822148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1590.830887] Lustre: Skipped 4 previous similar messages [ 1592.086749] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.59@tcp (stopping) [ 1592.099474] Lustre: Skipped 1 previous similar message [ 1593.945321] Lustre: server umount lustre-MDT0000 complete [ 1596.906510] LustreError: 51097:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1596.928330] LustreError: 51097:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1597.203146] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1603.043814] LustreError: 50370:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1603.064777] LustreError: 50370:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1603.133470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1603.234628] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1603.456834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1603.507020] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1607.132734] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1609.714964] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1610.234991] Lustre: DEBUG MARKER: Wait for df (36647) ... [ 1611.656509] Lustre: DEBUG MARKER: done [ 1615.331222] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1615.338538] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1615.367929] Lustre: Skipped 2 previous similar messages [ 1615.377177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1618.917182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1618.928677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1619.944790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1619.950230] Lustre: Skipped 2 previous similar messages [ 1621.269784] Lustre: server umount lustre-OST0000 complete [ 1625.057264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1625.065331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1625.091116] Lustre: Skipped 1 previous similar message [ 1630.795828] Lustre: server umount lustre-MDT0000 complete [ 1631.205931] LustreError: 50369:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1631.226688] LustreError: 50369:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1634.361262] LustreError: 50352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789597691 with bad export cookie 202885615836557158 [ 1634.374323] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1634.690493] Lustre: server umount lustre-MDT0001 complete [ 1644.138220] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 1646.704651] Key type lgssc unregistered [ 1647.016746] LNet: 55288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1647.028497] LNetError: 55288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1647.041073] LNet: Removed LNI 192.168.206.159@tcp [ 1647.769065] Key type .llcrypt unregistered [ 1647.771651] Key type ._llcrypt unregistered [ 1683.886294] Key type ._llcrypt registered [ 1683.888415] Key type .llcrypt registered [ 1683.982855] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 1684.975322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1685.070145] alg: No test for adler32 (adler32-zlib) [ 1686.145737] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 1686.415301] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 1688.160191] Key type lgssc registered [ 1689.265395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1698.178426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1698.202402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1699.514151] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1699.567847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1699.778862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1704.185598] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1712.934324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1713.092474] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1713.129984] Lustre: 57474:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1713.434082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1718.853752] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1727.042761] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1734.582873] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1741.503333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1741.637196] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1741.783763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1747.878958] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1750.632083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1754.037979] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1760.886679] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-1618h30m17s [ 1762.395705] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1768.841373] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1771.705942] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1771.771699] LustreError: 57480:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1792.269313] Lustre: lustre-MDT0000: Client 0ecee66b-6447-4fd2-9d3e-6380171aec1e (at 192.168.206.59@tcp) reconnecting [ 1811.872205] LustreError: 57480:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1813.854950] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1816.848588] Lustre: DEBUG MARKER: done [ 1827.301845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1827.313283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1827.321313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1827.809339] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1827.821920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1827.844362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1830.154698] Lustre: server umount lustre-OST0000 complete [ 1836.517554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1836.529977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1836.550616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1840.488141] Lustre: server umount lustre-MDT0000 complete [ 1843.734102] LustreError: 60236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789597900 with bad export cookie 14411237625650772089 [ 1843.742863] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1843.748153] LustreError: 60236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1844.234242] Lustre: server umount lustre-MDT0001 complete [ 1856.432973] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 1858.973405] Key type lgssc unregistered [ 1859.274560] LNet: 61804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1859.283830] LNetError: 61804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1859.302334] LNet: Removed LNI 192.168.206.159@tcp [ 1860.254737] Key type .llcrypt unregistered [ 1860.257222] Key type ._llcrypt unregistered [ 1902.446596] Key type ._llcrypt registered [ 1902.451462] Key type .llcrypt registered [ 1902.572411] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 1903.762871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1903.821914] alg: No test for adler32 (adler32-zlib) [ 1905.055800] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 1905.370911] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 1907.032194] Key type lgssc registered [ 1908.067505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1911.455166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1911.486636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1912.831523] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1912.869515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1913.084309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1914.089829] Lustre: Failing over lustre-MDT0000 [ 1914.361156] Lustre: server umount lustre-MDT0000 complete [ 1921.158564] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 18:32:57 (1789597977) [ 1927.171253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1927.709685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1932.349722] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1940.709265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1940.823975] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1940.865560] Lustre: 64579:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1941.132863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1945.469468] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1953.625515] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1960.049732] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1967.146082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1967.347672] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1967.621486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1974.194627] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 1975.981613] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1982.238052] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1988.284895] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1996.771685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1996.775129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1996.784662] Lustre: Skipped 2 previous similar messages [ 1996.793113] Lustre: Skipped 1 previous similar message [ 2000.938977] Lustre: server umount lustre-MDT0000 complete [ 2004.816300] LustreError: 64572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789598061 with bad export cookie 14772627596436976952 [ 2004.820361] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2004.830455] LustreError: 64572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2005.143740] Lustre: server umount lustre-MDT0001 complete [ 2009.008720] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2014.215530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2023.392328] Lustre: 63500:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789598064/real 1789598064] req@000000007865c777 x1876529472642560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789598080 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2023.418714] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2024.033512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2025.440350] Lustre: 63503:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789598066/real 1789598066] req@00000000e96301f5 x1876529472642688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789598082 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2028.513597] Lustre: 63503:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789598069/real 1789598069] req@000000009b7e5941 x1876529472642816/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789598085 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2029.984230] LustreError: 66427:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2030.015224] LustreError: 66427:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2030.165222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2030.330617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2031.136243] Lustre: 63501:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789598071/real 1789598071] req@00000000c6b6aaa2 x1876529472643072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789598087 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2032.174722] LustreError: 66428:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2032.190050] LustreError: 66428:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2033.185807] LustreError: 68350:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2033.203173] LustreError: 68350:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2035.273089] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2036.192142] Lustre: 63500:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789598076/real 1789598076] req@000000006478c30c x1876529472643200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789598092 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2038.244067] LustreError: 66427:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2038.274479] LustreError: 66427:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2043.362916] LustreError: 68349:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2043.383554] LustreError: 68349:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2044.777382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2045.107231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2050.109153] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2050.533656] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2052.593644] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2057.645569] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2063.746887] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2070.985100] Lustre: server umount lustre-MDT0000 complete [ 2071.009767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2071.030394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.051338] LustreError: 66428:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2073.066263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2073.086648] Lustre: Skipped 1 previous similar message [ 2075.171302] LustreError: 64570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789598132 with bad export cookie 14772627596436979227 [ 2075.191973] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2075.199592] LustreError: 64570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2075.449187] Lustre: server umount lustre-MDT0001 complete [ 2081.817070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2092.446835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2094.560166] Lustre: 63503:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789598135/real 1789598135] req@000000005522d019 x1876529472670848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789598151 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2094.590245] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2100.768942] LustreError: 63499:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ac3042bc x1876529472672000/t0(0) o250->MGC192.168.206.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2101.257602] LustreError: 66427:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2101.281456] LustreError: 66427:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2101.359889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2101.456958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2105.959937] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2113.009364] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2114.664714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2119.376589] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2120.177639] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2126.941244] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2133.616745] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2140.064196] Lustre: server umount lustre-MDT0000 complete [ 2140.645933] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2140.646918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2140.652972] LustreError: 69362:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2140.652982] LustreError: 69362:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 2140.664968] Lustre: Skipped 1 previous similar message [ 2144.131057] LustreError: 64571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789598201 with bad export cookie 14772627596436980606 [ 2144.134133] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2144.142957] LustreError: 64571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2144.482789] Lustre: server umount lustre-MDT0001 complete [ 2151.090499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2154.868451] Lustre: DEBUG MARKER: files compared the same [ 2166.299641] Lustre: server umount lustre-OST0000 complete [ 2182.417406] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 2185.022844] Key type lgssc unregistered [ 2185.327930] LNet: 74307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2185.338307] LNetError: 74307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2185.351581] LNet: Removed LNI 192.168.206.159@tcp [ 2186.386478] Key type .llcrypt unregistered [ 2186.389529] Key type ._llcrypt unregistered [ 2216.436601] Key type ._llcrypt registered [ 2216.446251] Key type .llcrypt registered [ 2216.578234] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 2217.212972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2217.274948] alg: No test for adler32 (adler32-zlib) [ 2218.208872] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 2218.378054] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 2220.088333] Key type lgssc registered [ 2221.111061] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2229.194573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2229.212082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2230.701370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2235.748844] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug +malloc all [ 2240.994125] LustreError: 75723:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2241.022943] LustreError: 75723:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2244.201668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2244.433594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2248.105459] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug +malloc all [ 2257.180272] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2263.576262] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2271.153275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2271.719063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2276.863745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2278.182669] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug +malloc all [ 2285.788185] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2297.314222] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2297.317921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2297.336921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2299.361428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2299.362275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2299.367203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2299.391390] Lustre: Skipped 1 previous similar message [ 2303.448231] Lustre: server umount lustre-OST0000 complete [ 2309.602090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2309.605523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2309.613963] Lustre: Skipped 1 previous similar message [ 2309.632722] Lustre: Skipped 1 previous similar message [ 2312.632819] Lustre: server umount lustre-MDT0000 complete [ 2314.720982] LustreError: 75723:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.731440] LustreError: 75723:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2315.604099] LustreError: 75703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789598372 with bad export cookie 17803597138487997977 [ 2315.605803] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2315.615240] LustreError: 75703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2315.855393] Lustre: server umount lustre-MDT0001 complete [ 2325.751930] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 2328.481582] Key type lgssc unregistered [ 2328.782406] LNet: 79252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2328.798420] LNetError: 79252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2328.814726] LNet: Removed LNI 192.168.206.159@tcp [ 2329.618550] Key type .llcrypt unregistered [ 2329.621463] Key type ._llcrypt unregistered [ 2352.386626] Key type ._llcrypt registered [ 2352.389643] Key type .llcrypt registered [ 2352.472643] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 2353.536339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2353.547823] alg: No test for adler32 (adler32-zlib) [ 2354.563607] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 2354.835770] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 2356.512281] Key type lgssc registered [ 2357.608995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2364.955394] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2364.972441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2380.512369] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2385.632322] LustreError: 80641:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.159@tcp: failed processing log, type 4: rc = -110 [ 2412.256179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2418.449270] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2425.533598] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2432.273143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2432.373966] Lustre: *** cfs_fail_loc=706, val=0*** [ 2432.380843] LustreError: 79976:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d3f80283 x1876529943415040/t0(0) o250->MGC192.168.206.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2432.672201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2432.756600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2436.698341] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2438.115570] LustreError: 80666:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2438.142921] LustreError: 80666:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2443.235262] LustreError: 80665:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2444.039926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2444.519062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2449.181439] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2457.427632] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2463.454556] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2468.326470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2468.336797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2468.352640] Lustre: Skipped 1 previous similar message [ 2468.371533] Lustre: Skipped 1 previous similar message [ 2471.927941] Lustre: server umount lustre-OST0000 complete [ 2478.565249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2478.576285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2478.585022] Lustre: Skipped 1 previous similar message [ 2478.607130] Lustre: Skipped 1 previous similar message [ 2483.119572] Lustre: server umount lustre-MDT0000 complete [ 2483.686492] LustreError: 81730:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2483.706658] LustreError: 81730:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2487.390506] LustreError: 82466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789598544 with bad export cookie 5359486511128871764 [ 2487.398499] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2487.402907] LustreError: 82466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2488.032986] Lustre: server umount lustre-MDT0001 complete [ 2500.042473] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 2503.599308] Key type lgssc unregistered [ 2503.969989] LNet: 84253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2503.986892] LNetError: 84253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2504.022964] LNet: Removed LNI 192.168.206.159@tcp [ 2504.985828] Key type .llcrypt unregistered [ 2504.991606] Key type ._llcrypt unregistered [ 2531.462396] Key type ._llcrypt registered [ 2531.465672] Key type .llcrypt registered [ 2531.617407] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 2532.883535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2533.055678] alg: No test for adler32 (adler32-zlib) [ 2534.162789] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 2534.510851] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 2536.211090] Key type lgssc registered [ 2537.785816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2546.342740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2546.383553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2553.320364] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2561.726475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2562.260571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2566.733334] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2567.653399] LustreError: 86356:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.678021] LustreError: 86356:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2572.777817] LustreError: 86355:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2572.803835] LustreError: 86355:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2575.644825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2575.934321] LustreError: 86356:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.956560] LustreError: 86356:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2575.965456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2580.962437] LustreError: 86360:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.978428] LustreError: 86360:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2581.575558] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2586.082648] LustreError: 86359:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2586.090946] LustreError: 86359:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2589.116806] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2589.132901] Lustre: MGS: Client ee064a41-4067-4ee0-b165-0ea894147465 (at 0@lo) reconnecting [ 2589.158300] LustreError: 86370:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2589.412820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2589.492202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2593.767531] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2610.150448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2610.166809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2610.177810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2610.187072] Lustre: Skipped 1 previous similar message [ 2611.787382] Lustre: server umount lustre-OST0000 complete [ 2615.266671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2615.268813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2615.275578] Lustre: Skipped 2 previous similar messages [ 2615.286437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2620.389559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2620.393342] Lustre: Skipped 2 previous similar messages [ 2621.424555] Lustre: server umount lustre-MDT0000 complete [ 2626.031247] Lustre: server umount lustre-MDT0001 complete [ 2629.831281] Lustre: server umount MGS complete [ 2641.052288] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 18:44:57 (1789598697) [ 2670.399112] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 2680.463767] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 2715.451239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2720.306750] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2730.523572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2730.728707] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2730.771519] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2730.847283] Lustre: lustre-MDT0001: new disk, initializing [ 2730.954039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2730.989255] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2731.006045] Lustre: 93524:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2731.027232] LustreError: 93524:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2735.117208] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2746.206410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2746.548414] Lustre: lustre-OST0000: new disk, initializing [ 2746.552550] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2746.558094] Lustre: 94291:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2746.635957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2752.952780] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2753.131662] Lustre: 94580:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000010eab35e x1876530132195712/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789598826 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2763.227828] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2770.299849] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2770.316965] Lustre: MGS: Client bd27e570-11b7-4c83-b736-58bcd735fbba (at 0@lo) reconnecting [ 2770.403887] Lustre: 92789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2770.425692] LustreError: 93509:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2770.441714] LustreError: 93509:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2770.459294] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2770.518847] Lustre: lustre-MDT0000: new disk, initializing [ 2770.655889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2770.681659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2770.702526] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2771.746861] Lustre: 95359:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000932f618b x1876530132202624/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789598844 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2776.786815] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2776.867495] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2780.655032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2780.678967] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2780.914844] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2801.640096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2801.668648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2801.680123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2803.683761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2807.660520] Lustre: server umount lustre-OST0000 complete [ 2813.921826] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2813.927794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2813.937449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2813.955578] Lustre: Skipped 3 previous similar messages [ 2817.206878] Lustre: server umount lustre-MDT0000 complete [ 2819.042158] LustreError: 95643:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2819.060170] LustreError: 95643:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2827.476291] Lustre: server umount lustre-MDT0001 complete [ 2831.136239] Lustre: server umount MGS complete [ 2843.091515] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 18:48:19 (1789598899) [ 2862.212266] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 2865.352805] Key type lgssc unregistered [ 2865.619356] LNet: 97859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2865.636186] LNetError: 97859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2865.651754] LNet: Removed LNI 192.168.206.159@tcp [ 2866.644903] Key type .llcrypt unregistered [ 2866.651773] Key type ._llcrypt unregistered [ 2884.196385] Key type ._llcrypt registered [ 2884.205277] Key type .llcrypt registered [ 2884.380681] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 2885.256914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2885.350080] alg: No test for adler32 (adler32-zlib) [ 2886.432562] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 2886.680396] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 2888.336184] Key type lgssc registered [ 2889.227871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2895.811451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2895.830782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2897.166482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2901.340462] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2907.620414] LustreError: 99080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2907.637351] LustreError: 99080:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2911.175256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2911.557661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2917.847496] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2925.688267] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2931.841511] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2938.553275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2939.020621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2942.076796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2945.194516] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 2950.920218] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2961.890386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2961.906017] Lustre: Skipped 1 previous similar message [ 2961.906142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2961.919820] Lustre: Skipped 1 previous similar message [ 2963.823662] Lustre: server umount lustre-OST0000 complete [ 2968.035242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2968.052691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2968.072086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2972.130350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2972.143527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2973.421456] Lustre: server umount lustre-MDT0000 complete [ 2976.565401] LustreError: 99059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789599033 with bad export cookie 10744347225711073615 [ 2976.567408] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2976.574877] LustreError: 99059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2976.986357] Lustre: server umount lustre-MDT0001 complete [ 2987.007842] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 2989.490508] Key type lgssc unregistered [ 2989.861781] LNet: 102615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2989.873320] LNetError: 102615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2989.890989] LNet: Removed LNI 192.168.206.159@tcp [ 2991.049749] Key type .llcrypt unregistered [ 2991.056865] Key type ._llcrypt unregistered [ 3008.113210] Key type ._llcrypt registered [ 3008.116499] Key type .llcrypt registered [ 3008.261108] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 3009.523440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3009.810198] alg: No test for adler32 (adler32-zlib) [ 3010.975389] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 3011.264588] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 3012.936364] Key type lgssc registered [ 3014.170636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3020.408068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3020.417392] LustreError: 103702:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3020.443199] LustreError: 103706:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3020.448314] LustreError: 103702:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4979 [ 3020.451324] LustreError: 103702:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3020.458625] LustreError: 103702:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3020.474576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3022.029852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3029.520119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3029.819813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3034.677384] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3040.066900] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing lsmod [ 3042.802414] LustreError: 104904:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3042.852886] LustreError: 104911:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3042.857337] LustreError: 104904:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4952 [ 3042.865684] LustreError: 104904:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3042.871371] LustreError: 104904:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3042.877398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3043.201993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3048.420926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3048.422514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3048.440151] Lustre: Skipped 1 previous similar message [ 3048.462858] Lustre: Skipped 2 previous similar messages [ 3052.662895] Lustre: server umount lustre-MDT0000 complete [ 3053.543490] LustreError: 103743:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3053.577765] LustreError: 103743:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3054.562769] LustreError: 103743:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3054.585973] LustreError: 103743:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3056.716588] LustreError: 103719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789599113 with bad export cookie 13374286004501075797 [ 3056.733680] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3056.745658] LustreError: 103719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3058.723792] LustreError: 104174:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000018d189b9 x1876530632347136/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3058.926988] Lustre: server umount lustre-MDT0001 complete [ 3073.457494] Lustre: server umount lustre-OST0000 complete [ 3081.709918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3082.198705] LustreError: 105975:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3082.269400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3087.272403] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3087.337155] LustreError: 105976:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3092.453459] LustreError: 105975:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3097.177488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3097.505365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3101.789041] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3109.482734] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3116.261568] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3126.010282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3126.335533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3127.424141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3133.379256] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3141.554966] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3148.257213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3148.263041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3148.269137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3148.285823] Lustre: Skipped 2 previous similar messages [ 3148.291789] Lustre: Skipped 1 previous similar message [ 3153.379135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3154.338415] Lustre: server umount lustre-OST0000 complete [ 3159.009585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3159.019722] LustreError: Skipped 1 previous similar message [ 3159.025715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3159.048833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3159.050930] Lustre: Skipped 1 previous similar message [ 3163.618512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3163.631857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3173.344406] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3173.635114] Lustre: server umount lustre-MDT0000 complete [ 3173.866083] LustreError: 105971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3173.887658] LustreError: 105971:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3177.021805] LustreError: 105955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789599234 with bad export cookie 13374286004501076483 [ 3177.029337] LustreError: 105955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3177.041332] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3177.387681] Lustre: server umount lustre-MDT0001 complete [ 3187.580639] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 3190.471351] Key type lgssc unregistered [ 3190.806218] LNet: 109509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3190.823186] LNetError: 109509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3190.850972] LNet: Removed LNI 192.168.206.159@tcp [ 3191.686532] Key type .llcrypt unregistered [ 3191.689762] Key type ._llcrypt unregistered [ 3215.853844] Key type ._llcrypt registered [ 3215.860662] Key type .llcrypt registered [ 3215.959752] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 3216.907762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3216.968890] alg: No test for adler32 (adler32-zlib) [ 3217.989761] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 3218.167599] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 3219.808216] Key type lgssc registered [ 3220.816274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3227.404120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3227.420598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3228.822129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3232.218351] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3238.371548] LustreError: 110923:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3238.386793] LustreError: 110923:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3239.151718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3239.506940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3243.498192] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3250.542090] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3255.998626] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3263.896272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3264.206252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3270.278427] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3273.504104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3277.841293] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3283.727274] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3286.625267] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3291.112454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3291.132679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3291.150960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3292.527660] Lustre: server umount lustre-OST0000 complete [ 3299.301519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3299.310648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3299.314261] Lustre: Skipped 1 previous similar message [ 3304.417732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3304.424210] Lustre: Skipped 2 previous similar messages [ 3306.556598] Lustre: server umount lustre-MDT0000 complete [ 3309.538179] LustreError: 110918:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3309.566886] LustreError: 110918:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3310.139655] LustreError: 110903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789599367 with bad export cookie 13329222923285320006 [ 3310.140247] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3310.152506] LustreError: 110903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3310.623713] Lustre: server umount lustre-MDT0001 complete [ 3320.893202] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 3323.580228] Key type lgssc unregistered [ 3323.849026] LNet: 114554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3323.861240] LNetError: 114554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3323.879954] LNet: Removed LNI 192.168.206.159@tcp [ 3324.756871] Key type .llcrypt unregistered [ 3324.759461] Key type ._llcrypt unregistered [ 3341.678311] Key type ._llcrypt registered [ 3341.682657] Key type .llcrypt registered [ 3341.767590] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 3342.933729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3342.966175] alg: No test for adler32 (adler32-zlib) [ 3344.106337] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 3344.383195] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 3346.057242] Key type lgssc registered [ 3347.245216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3355.632819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3355.642693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3357.085609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3362.760574] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3367.402678] LustreError: 115775:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3367.425527] LustreError: 115775:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3368.431862] LustreError: 115774:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3373.086523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3373.541624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3378.310663] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3387.248586] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3393.787525] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3401.207482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3401.508592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3403.566332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3407.694440] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3415.179134] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3427.822545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3427.827659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.841284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3429.345493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3429.350421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3429.360594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3429.375228] Lustre: Skipped 1 previous similar message [ 3433.510507] Lustre: server umount lustre-OST0000 complete [ 3439.586954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3439.592286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3439.611380] Lustre: Skipped 1 previous similar message [ 3439.625706] Lustre: Skipped 1 previous similar message [ 3443.854701] Lustre: server umount lustre-MDT0000 complete [ 3444.707736] LustreError: 118267:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3444.725450] LustreError: 118267:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3447.848593] LustreError: 115755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789599504 with bad export cookie 7500161871810770513 [ 3447.863409] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3447.866380] LustreError: 115755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3448.252787] Lustre: server umount lustre-MDT0001 complete [ 3458.875523] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 3461.919633] Key type lgssc unregistered [ 3462.266761] LNet: 119325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3462.280287] LNetError: 119325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3462.299311] LNet: Removed LNI 192.168.206.159@tcp [ 3463.196735] Key type .llcrypt unregistered [ 3463.201481] Key type ._llcrypt unregistered [ 3487.788918] Key type ._llcrypt registered [ 3487.790655] Key type .llcrypt registered [ 3487.950478] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 3489.124438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3489.155583] alg: No test for adler32 (adler32-zlib) [ 3490.362582] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 3490.565226] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 3492.264854] Key type lgssc registered [ 3493.491503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3502.165567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3502.181282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3503.525070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3507.757957] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3513.854526] LustreError: 120738:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3513.884952] LustreError: 120738:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3517.417072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3517.938354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3523.594620] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3533.839831] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3540.216259] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3547.571547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3547.822149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3548.846102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3553.370263] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3561.758744] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3569.518024] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3574.067568] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3574.072855] Lustre: Skipped 2 previous similar messages [ 3578.928600] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3578.946514] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3579.019680] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3579.024305] Lustre: Skipped 2 previous similar messages [ 3585.209296] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3585.235776] Lustre: Skipped 1 previous similar message [ 3589.739428] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.59@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3589.746580] Lustre: Skipped 2 previous similar messages [ 3593.421352] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.59@tcp 192.168.0.[2,10]@tcp [ 3598.219099] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.59/24@tcp in log lustre-MDT0000 [ 3598.236790] Lustre: Skipped 2 previous similar messages [ 3603.024275] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.59/24@tcp [ 3603.028866] Lustre: Skipped 1 previous similar message [ 3613.322361] Lustre: server umount lustre-OST0000 complete [ 3618.787346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3618.792674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3618.805165] Lustre: Skipped 1 previous similar message [ 3618.821724] Lustre: Skipped 1 previous similar message [ 3623.906310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3623.913984] Lustre: Skipped 1 previous similar message [ 3628.131251] Lustre: server umount lustre-MDT0000 complete [ 3629.024288] Lustre: 120047:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789599670/real 1789599670] req@000000009b880e98 x1876531134661760/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789599686 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3629.028483] LustreError: 120734:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3629.071434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3629.089567] LustreError: 120734:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3630.048161] Lustre: 120048:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789599670/real 1789599670] req@000000006478c30c x1876531134661376/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789599686 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3632.018511] LustreError: 120718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789599689 with bad export cookie 15185466764000033972 [ 3632.020105] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3632.032012] LustreError: 120718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3632.430829] Lustre: server umount lustre-MDT0001 complete [ 3645.355742] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 3648.667876] Key type lgssc unregistered [ 3649.191704] LNet: 125695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3649.204245] LNetError: 125695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3649.228360] LNet: Removed LNI 192.168.206.159@tcp [ 3650.245623] Key type .llcrypt unregistered [ 3650.248062] Key type ._llcrypt unregistered [ 3679.576176] Key type ._llcrypt registered [ 3679.582493] Key type .llcrypt registered [ 3679.729924] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 3680.684996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3680.811672] alg: No test for adler32 (adler32-zlib) [ 3681.919869] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 3682.157642] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 3683.864266] Key type lgssc registered [ 3685.136871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3693.484001] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3693.500827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3694.951252] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3694.986229] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.59/24@tcp [ 3694.999088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3699.420445] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3704.293147] LustreError: 127251:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3704.305900] LustreError: 127251:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3707.251547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3707.572070] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3707.611775] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.59/24@tcp [ 3707.672775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3711.684422] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3718.908520] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3725.047816] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3731.880962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3732.249686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3737.680880] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3739.455374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3745.174893] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3754.982329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3754.989793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3755.001078] Lustre: Skipped 1 previous similar message [ 3755.015133] Lustre: Skipped 1 previous similar message [ 3760.102909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3760.111680] Lustre: Skipped 1 previous similar message [ 3765.222437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3765.228166] Lustre: Skipped 1 previous similar message [ 3769.318024] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3769.562477] Lustre: server umount lustre-OST0000 complete [ 3774.434409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3774.443140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3774.466582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3775.460825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3780.599132] LustreError: 127251:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3780.663935] Lustre: server umount lustre-MDT0000 complete [ 3784.816438] LustreError: 127234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789599841 with bad export cookie 10837784404240671093 [ 3784.820641] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3784.829052] LustreError: 127234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3785.338373] Lustre: server umount lustre-MDT0001 complete [ 3797.455850] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 3800.254948] Key type lgssc unregistered [ 3800.537179] LNet: 130839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3800.544552] LNetError: 130839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3800.565283] LNet: Removed LNI 192.168.206.159@tcp [ 3801.483836] Key type .llcrypt unregistered [ 3801.486702] Key type ._llcrypt unregistered [ 3827.891855] Key type ._llcrypt registered [ 3827.897434] Key type .llcrypt registered [ 3828.025702] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 3842.675430] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 3843.859402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3843.948131] alg: No test for adler32 (adler32-zlib) [ 3845.054442] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 3845.385923] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 3847.113477] Key type lgssc registered [ 3848.381418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3886.396276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3886.420722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3887.823656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3887.858905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3887.993383] Lustre: lustre-MDT0000: new disk, initializing [ 3888.057657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3888.071245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3893.141776] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3903.798979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3903.935990] Lustre: 134235:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3904.012281] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3904.018609] Lustre: Skipped 1 previous similar message [ 3904.093306] Lustre: lustre-MDT0001: new disk, initializing [ 3904.185842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3904.213046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3904.230701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3908.600930] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3918.145822] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3923.756678] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3931.007812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3931.226818] Lustre: lustre-OST0000: new disk, initializing [ 3931.229854] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3931.240729] Lustre: 136202:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3931.381651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3937.379492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3937.409575] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3937.504067] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3938.145787] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 3947.662517] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3951.864966] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3952.011849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3956.662058] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3956.860756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3963.364666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3963.365173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3963.384986] LustreError: Skipped 1 previous similar message [ 3963.390863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3963.415397] Lustre: Skipped 1 previous similar message [ 3963.439059] Lustre: Skipped 1 previous similar message [ 3968.488309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3968.497112] Lustre: Skipped 1 previous similar message [ 3973.088123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3973.338943] Lustre: server umount lustre-OST0000 complete [ 3978.722353] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3978.742312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3978.750867] Lustre: Skipped 1 previous similar message [ 3983.837545] Lustre: server umount lustre-MDT0000 complete [ 3987.566753] LustreError: 134227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789600044 with bad export cookie 15906144983252914422 [ 3987.571102] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3987.583027] LustreError: 134227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3987.972985] Lustre: server umount lustre-MDT0001 complete [ 3996.477668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3997.013444] LustreError: 138422:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3997.031991] LustreError: 138422:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3997.084990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4001.488196] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 4002.273658] LustreError: 138423:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4007.395648] LustreError: 138422:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4009.872780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4010.249650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4014.714582] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 4022.386888] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4028.827462] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4035.894798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4036.329633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4043.013763] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 4050.423990] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4054.856450] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4055.082560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4059.679359] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4059.882107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4068.101877] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4068.310185] Lustre: lustre-OST0001: new disk, initializing [ 4068.314657] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4068.322549] Lustre: 141793:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4068.395507] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4074.431759] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 4075.151782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4075.160139] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4075.262844] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4082.765949] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4083.033104] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4087.712690] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4087.966274] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4093.109636] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93c7e387a800.ost_server_uuid 50 [ 4094.759531] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff93c7e387a800.ost_server_uuid in FULL state after 0 sec [ 4106.218405] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4106.219177] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4106.220610] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4106.232276] Lustre: Skipped 2 previous similar messages [ 4106.259954] LustreError: Skipped 1 previous similar message [ 4107.920825] Lustre: server umount lustre-OST0001 complete [ 4111.343073] LustreError: 140244:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4111.360651] LustreError: 140244:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4112.355737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4112.362783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4112.390413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4112.397723] Lustre: Skipped 1 previous similar message [ 4114.402656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4116.462633] LustreError: 140245:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4116.477773] LustreError: 140245:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4121.577051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4121.585459] Lustre: Skipped 4 previous similar messages [ 4126.176345] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4126.310214] Lustre: server umount lustre-OST0000 complete [ 4131.822372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4131.837902] Lustre: Skipped 2 previous similar messages [ 4135.932868] Lustre: server umount lustre-MDT0000 complete [ 4136.929531] LustreError: 138419:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4136.945552] LustreError: 138419:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4139.267591] LustreError: 138403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789600196 with bad export cookie 15906144983252915304 [ 4139.271724] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4139.280957] LustreError: 138403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4139.604702] Lustre: server umount lustre-MDT0001 complete [ 4157.146219] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 4159.701903] Key type lgssc unregistered [ 4159.935586] LNet: 144796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4159.941725] LNetError: 144796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4159.962446] LNet: Removed LNI 192.168.206.159@tcp [ 4160.646048] Key type .llcrypt unregistered [ 4160.648865] Key type ._llcrypt unregistered [ 4195.052428] Key type ._llcrypt registered [ 4195.054554] Key type .llcrypt registered [ 4195.153446] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 4208.935223] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 4210.225025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4210.315860] alg: No test for adler32 (adler32-zlib) [ 4211.340640] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 4211.530831] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 4213.184439] Key type lgssc registered [ 4214.296905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4250.221740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4250.241242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4251.490660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4251.520635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4251.619744] Lustre: lustre-MDT0000: new disk, initializing [ 4251.730307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4251.755657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4255.762674] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 4264.926397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4265.062288] Lustre: 148948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4265.105317] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4265.111740] Lustre: Skipped 1 previous similar message [ 4265.164663] Lustre: lustre-MDT0001: new disk, initializing [ 4265.282475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4265.335596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4265.348890] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4269.317947] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 4277.761046] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4282.880035] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4289.108461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4289.425532] Lustre: lustre-OST0000: new disk, initializing [ 4289.435753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4289.453221] Lustre: 150918:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4289.588992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4296.148789] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 4297.301029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4297.305970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4297.442298] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4304.789444] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4309.563789] Lustre: Failing over lustre-OST0000 [ 4309.674418] Lustre: server umount lustre-OST0000 complete [ 4327.975826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4328.324184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4328.365989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4328.394900] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4328.928122] Lustre: 147170:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789600369/real 1789600369] req@000000008f6711f4 x1876531890650624/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789600385 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4329.440726] Lustre: 147167:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789600370/real 1789600370] req@000000005d8d95c8 x1876531890651520/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789600386 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4329.471163] Lustre: 147167:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4333.263618] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4333.561193] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4333.564868] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4333.728314] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 4334.626093] Lustre: 147168:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789600375/real 1789600375] req@00000000e62996ce x1876531890652928/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789600391 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4334.654201] Lustre: 147168:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4337.436671] Lustre: Failing over lustre-MDT0000 [ 4337.778631] Lustre: server umount lustre-MDT0000 complete [ 4338.661609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4338.688260] Lustre: Skipped 2 previous similar messages [ 4339.744105] Lustre: 147167:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789600380/real 1789600380] req@0000000032c710fd x1876531890653952/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789600396 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4343.518384] LustreError: 148955:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4343.541722] LustreError: 148955:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4348.622824] LustreError: 148953:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4348.643625] LustreError: 148953:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4353.751369] LustreError: 148953:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4353.785739] LustreError: 148953:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4354.978485] Lustre: 147170:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789600395/real 1789600395] req@000000007df77a8c x1876531890662656/t0(0) o400->MGC192.168.206.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789600411 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4355.008672] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4358.519791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4358.673610] LustreError: 148954:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4358.692130] LustreError: 148954:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4364.822549] LustreError: 148955:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4364.860807] LustreError: 148955:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4365.298840] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4c64bbd3b22a3f52 [ 4365.304766] Lustre: MGC192.168.206.159@tcp: Connection restored to 0@lo (at 0@lo) [ 4365.315774] Lustre: Skipped 1 previous similar message [ 4365.551226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4365.600540] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4369.837608] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 4370.134765] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4370.919875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4370.964594] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4371.007231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4379.619830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4379.633143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4379.651793] Lustre: Skipped 1 previous similar message [ 4379.658387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4381.161328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4381.171213] Lustre: Skipped 1 previous similar message [ 4382.806771] Lustre: server umount lustre-OST0000 complete [ 4386.276357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4386.277439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4386.281185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4386.292788] Lustre: Skipped 2 previous similar messages [ 4391.393512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4391.404661] Lustre: Skipped 2 previous similar messages [ 4392.348217] Lustre: server umount lustre-MDT0000 complete [ 4395.568611] LustreError: 150917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789600452 with bad export cookie 5504731162455195474 [ 4395.578552] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4395.581033] LustreError: 150917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4396.007664] Lustre: server umount lustre-MDT0001 complete [ 4405.935338] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 4408.327982] Key type lgssc unregistered [ 4408.566883] LNet: 154574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4408.577252] LNetError: 154574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4408.596850] LNet: Removed LNI 192.168.206.159@tcp [ 4409.429503] Key type .llcrypt unregistered [ 4409.431533] Key type ._llcrypt unregistered [ 4446.522710] Key type ._llcrypt registered [ 4446.524326] Key type .llcrypt registered [ 4446.688152] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 4447.230684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4447.301067] alg: No test for adler32 (adler32-zlib) [ 4448.340423] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 4448.539470] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 4450.216486] Key type lgssc registered [ 4451.157913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4458.987257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4459.003917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4460.376565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4465.135876] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 4465.636698] LustreError: 156092:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4465.669699] LustreError: 156092:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4470.757598] LustreError: 156093:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4474.652758] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4475.885942] LustreError: 156092:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4478.887378] LustreError: 156121:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4478.903913] LustreError: 156121:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4478.909077] LustreError: 156121:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4479.179334] Lustre: server umount lustre-MDT0000 complete [ 4494.116879] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 4503.568754] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 4536.691987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4536.904672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4536.946943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4537.057368] Lustre: lustre-MDT0000: new disk, initializing [ 4537.143168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4537.159901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4541.091627] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 4550.132520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4550.256714] Lustre: 159432:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4550.310832] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4550.314133] Lustre: Skipped 1 previous similar message [ 4550.373079] Lustre: lustre-MDT0001: new disk, initializing [ 4550.465177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4550.534277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4550.549555] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4554.884651] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 4563.976433] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4569.953077] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4577.026872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4577.454931] Lustre: lustre-OST0000: new disk, initializing [ 4577.459755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4577.465658] Lustre: 161399:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4577.548949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4584.547205] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 4587.625359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4587.640604] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4587.812085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4593.785892] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4597.185692] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4599.314299] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 19:17:35 (1789600655) [ 4862.945061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4862.957114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4862.985360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4864.482332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4864.484040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4864.507019] Lustre: Skipped 1 previous similar message [ 4864.526986] Lustre: Skipped 2 previous similar messages [ 4869.603494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4869.608165] Lustre: Skipped 2 previous similar messages [ 4874.208328] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4874.344949] Lustre: server umount lustre-MDT0000 complete [ 4874.731370] LustreError: 159437:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4874.754980] LustreError: 159437:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4878.106491] LustreError: 159422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789600935 with bad export cookie 12128779572666596477 [ 4878.115191] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4878.120259] LustreError: 159422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4879.842266] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4879.843620] LustreError: 159438:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4879.866820] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4883.875026] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4884.498322] Lustre: server umount lustre-MDT0001 complete [ 4902.368513] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4902.560420] Lustre: server umount lustre-OST0000 complete [ 4912.860304] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 4921.897813] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 4956.523146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4956.719592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4956.750622] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4956.840664] Lustre: lustre-MDT0000: new disk, initializing [ 4956.919151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4956.929780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4961.105481] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 4969.934047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4970.017837] Lustre: 166063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4970.059351] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4970.062959] Lustre: Skipped 1 previous similar message [ 4970.125332] Lustre: lustre-MDT0001: new disk, initializing [ 4970.220576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4970.249485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4970.266807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4974.236513] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 4982.470796] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4988.079605] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4994.605912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4994.923768] Lustre: lustre-OST0000: new disk, initializing [ 4994.926863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4994.933943] Lustre: 168033:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4995.007616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5000.952984] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5003.363580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5003.379533] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5003.505775] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5010.746718] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5015.781903] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5016.014389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5020.605458] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5020.838433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5024.234127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5024.236580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5024.253049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5024.268111] Lustre: Skipped 1 previous similar message [ 5028.611834] Lustre: server umount lustre-OST0000 complete [ 5034.474125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5044.711201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5044.720093] Lustre: Skipped 6 previous similar messages [ 5047.264137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5047.462397] Lustre: server umount lustre-MDT0000 complete [ 5049.826117] LustreError: 166956:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5049.854775] LustreError: 166956:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5051.228332] LustreError: 166055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789601108 with bad export cookie 12128779572666755314 [ 5051.244147] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5051.249941] LustreError: 166055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5051.934725] Lustre: server umount lustre-MDT0001 complete [ 5061.597976] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:25:17 (1789601117) [ 5081.820643] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 5093.730885] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 5130.961427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5131.176933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5131.221166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5131.344988] Lustre: lustre-MDT0000: new disk, initializing [ 5131.435810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5131.448886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5135.784356] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5146.661829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5146.889461] Lustre: 172790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5147.276686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5151.940550] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5160.916629] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5166.406654] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5171.551282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5171.832981] Lustre: lustre-OST0000: new disk, initializing [ 5171.836772] Lustre: Skipped 1 previous similar message [ 5171.841835] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5171.846073] Lustre: Skipped 2 previous similar messages [ 5171.850712] Lustre: 174768:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5177.066884] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5177.425685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5177.433656] Lustre: Skipped 1 previous similar message [ 5177.436933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5177.544396] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5185.715869] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5193.187233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5193.188472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5193.203547] LustreError: Skipped 2 previous similar messages [ 5193.211510] Lustre: Skipped 1 previous similar message [ 5193.215778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5193.228425] Lustre: Skipped 1 previous similar message [ 5197.666977] Lustre: server umount lustre-OST0000 complete [ 5203.434421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5207.520521] Lustre: server umount lustre-MDT0000 complete [ 5208.544957] LustreError: 172803:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5208.558387] LustreError: 172803:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5210.563725] LustreError: 172782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789601267 with bad export cookie 12128779572666756189 [ 5210.565318] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5210.571360] LustreError: 172782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5210.912547] Lustre: server umount lustre-MDT0001 complete [ 5218.096996] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:27:54 (1789601274) [ 5232.883782] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 5241.420920] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 5286.031235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5286.272976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5286.311160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5286.431220] Lustre: lustre-MDT0000: new disk, initializing [ 5286.585089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5286.596602] Lustre: Skipped 2 previous similar messages [ 5286.638819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5291.500529] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5301.457545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5301.645869] Lustre: 179096:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5301.914279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5306.195672] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5314.763486] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5319.501386] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5325.258776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5325.531736] Lustre: 181071:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5331.446260] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5334.079634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5334.229845] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5340.727525] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5349.860306] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5349.883258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5349.900800] Lustre: Skipped 3 previous similar messages [ 5349.906494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5349.916960] Lustre: Skipped 2 previous similar messages [ 5354.319526] Lustre: server umount lustre-OST0000 complete [ 5358.053120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5358.066270] LustreError: Skipped 1 previous similar message [ 5372.389617] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5372.650630] Lustre: server umount lustre-MDT0000 complete [ 5375.458183] LustreError: 180516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5376.379942] LustreError: 179086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789601433 with bad export cookie 12128779572666757281 [ 5376.383117] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5376.388721] LustreError: 179086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5376.607212] Lustre: server umount lustre-MDT0001 complete [ 5389.404788] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 5391.939925] Key type lgssc unregistered [ 5392.206157] LNet: 182926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5392.210511] LNetError: 182926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5393.258849] LNet: Removed LNI 192.168.206.159@tcp [ 5394.001252] Key type .llcrypt unregistered [ 5394.005554] Key type ._llcrypt unregistered [ 5417.913716] Key type ._llcrypt registered [ 5417.915732] Key type .llcrypt registered [ 5417.992797] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 5418.838792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5418.925130] alg: No test for adler32 (adler32-zlib) [ 5419.924253] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 5420.096509] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 5421.712349] Key type lgssc registered [ 5422.881760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5430.416669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5430.446807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5432.032106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5435.821659] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5442.535233] LustreError: 184335:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5442.552767] LustreError: 184335:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5442.996056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5443.348832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5447.123565] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5454.449909] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5461.291824] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5468.224602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5468.734213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5475.097195] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5477.944143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5482.965459] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5508.577722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5508.583942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5508.600123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5508.609135] Lustre: Skipped 1 previous similar message [ 5508.625402] Lustre: Skipped 1 previous similar message [ 5512.135959] Lustre: server umount lustre-OST0000 complete [ 5518.818589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5518.834784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5518.845102] Lustre: Skipped 1 previous similar message [ 5522.543721] Lustre: server umount lustre-MDT0000 complete [ 5523.938178] LustreError: 186829:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5523.949861] LustreError: 186829:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5525.845313] LustreError: 185070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789601582 with bad export cookie 10838271548655589240 [ 5525.856254] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5525.864509] LustreError: 185070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5526.224341] Lustre: server umount lustre-MDT0001 complete [ 5536.443594] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 5539.298237] Key type lgssc unregistered [ 5539.599267] LNet: 187891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5539.606488] LNetError: 187891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5540.648086] LNet: Removed LNI 192.168.206.159@tcp [ 5541.674751] Key type .llcrypt unregistered [ 5541.679055] Key type ._llcrypt unregistered [ 5564.461705] Key type ._llcrypt registered [ 5564.463176] Key type .llcrypt registered [ 5564.582241] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 5566.369161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5566.415687] alg: No test for adler32 (adler32-zlib) [ 5567.549317] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 5567.861858] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 5569.608337] Key type lgssc registered [ 5570.870398] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5579.366598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5579.380680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5580.842128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5585.520530] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5590.011668] LustreError: 189305:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5590.043280] LustreError: 189305:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5594.421468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5594.832266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5599.632581] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5607.588611] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5613.918592] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5621.523616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5622.062498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5625.163796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5629.367192] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5636.862580] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5649.651887] Lustre: lustre-OST0000: Not available for connect from 192.168.206.59@tcp (stopping) [ 5649.899368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5649.926909] Lustre: Skipped 1 previous similar message [ 5649.936545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5654.769719] Lustre: lustre-OST0000: Not available for connect from 192.168.206.59@tcp (stopping) [ 5654.780625] Lustre: Skipped 2 previous similar messages [ 5659.859636] Lustre: lustre-OST0000: Not available for connect from 192.168.206.59@tcp (stopping) [ 5659.866721] Lustre: Skipped 2 previous similar messages [ 5661.152186] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5661.343753] Lustre: server umount lustre-OST0000 complete [ 5670.291812] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff93c7c71d0800.ost_server_uuid 50 [ 5671.972627] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5687.266668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5687.279152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5687.323043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5687.330152] Lustre: Skipped 2 previous similar messages [ 5690.404147] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5694.916845] Lustre: server umount lustre-MDT0000 complete [ 5695.458220] LustreError: 189299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5695.484229] LustreError: 189299:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5699.080491] LustreError: 192257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789601756 with bad export cookie 3144371272901502766 [ 5699.082248] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5699.098475] LustreError: 192257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5699.723137] Lustre: server umount lustre-MDT0001 complete [ 5707.332367] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 19:36:04 (1789601764) [ 5713.741792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5714.018611] LustreError: 193229:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5714.064222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5718.133851] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5719.521520] LustreError: 193230:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5724.641584] LustreError: 193229:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5727.726240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5728.020918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5732.698552] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5741.254193] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5747.547210] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5754.962722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5755.328489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5759.435460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5761.626748] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5769.779782] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5776.610956] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5776.859430] Lustre: lustre-OST0001: new disk, initializing [ 5776.862442] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5776.869696] Lustre: 196070:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5776.946181] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5783.521410] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5786.741379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5786.755534] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5786.855142] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5793.239863] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5797.356291] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5797.360350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5797.370661] Lustre: Skipped 1 previous similar message [ 5797.386989] Lustre: Skipped 3 previous similar messages [ 5802.332405] Lustre: server umount lustre-OST0000 complete [ 5802.471193] LustreError: 195052:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5802.494820] LustreError: 195052:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5808.903417] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5809.087919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5813.580429] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5813.798717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5819.625805] LustreError: 195050:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5819.647395] LustreError: 195050:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 5828.065725] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5828.076571] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5828.082694] LustreError: Skipped 1 previous similar message [ 5828.085680] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5828.085691] Lustre: Skipped 1 previous similar message [ 5828.128479] Lustre: Skipped 1 previous similar message [ 5833.410914] Lustre: server umount lustre-OST0001 complete [ 5838.307302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5843.218405] Lustre: server umount lustre-MDT0000 complete [ 5847.010021] LustreError: 196069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789601904 with bad export cookie 3144371272901503837 [ 5847.011632] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5847.020315] LustreError: 196069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5847.395399] Lustre: server umount lustre-MDT0001 complete [ 5870.940436] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 19:38:47 (1789601927) [ 5876.826225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5877.143174] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5877.167954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5877.359969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5881.752292] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5890.448400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5890.525626] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5890.577114] Lustre: 199508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5895.637953] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5905.959793] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5911.871554] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5918.680293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5918.821584] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5925.494086] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5927.584407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5934.113224] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5941.691901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5941.802538] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5942.090522] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5942.107639] Lustre: Skipped 2 previous similar messages [ 5948.093561] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5949.160150] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 5956.721857] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5962.998988] Lustre: lustre-OST0000: Not available for connect from 192.168.206.59@tcp (stopping) [ 5963.015216] Lustre: Skipped 5 previous similar messages [ 5963.241718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5963.246815] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5963.253809] Lustre: Skipped 1 previous similar message [ 5966.225227] Lustre: server umount lustre-OST0000 complete [ 5968.090614] LustreError: 201360:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5968.117681] LustreError: 201360:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 5983.719618] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5983.730753] LustreError: Skipped 1 previous similar message [ 5987.012426] Lustre: server umount lustre-OST0001 complete [ 5993.450235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5993.459761] LustreError: Skipped 1 previous similar message [ 5999.585808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5999.598824] Lustre: Skipped 9 previous similar messages [ 6005.216147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6005.431950] Lustre: server umount lustre-MDT0000 complete [ 6008.779954] LustreError: 199501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602065 with bad export cookie 3144371272901505244 [ 6008.782236] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6008.797677] LustreError: 199501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6009.086524] Lustre: server umount lustre-MDT0001 complete [ 6033.271775] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 19:41:29 (1789602089) [ 6050.843535] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 6061.056533] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 6092.681778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6092.864137] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6092.889617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6092.968235] Lustre: lustre-MDT0000: new disk, initializing [ 6093.058114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6093.069678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6097.189291] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6105.941111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6106.034922] Lustre: 207609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6106.090401] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6106.094663] Lustre: Skipped 1 previous similar message [ 6106.180123] Lustre: lustre-MDT0001: new disk, initializing [ 6106.302430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6106.313128] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6109.723413] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6118.796716] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6124.975670] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6131.408283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6131.604543] Lustre: lustre-OST0000: new disk, initializing [ 6131.609442] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6131.616486] Lustre: 209576:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6136.580857] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6141.503886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6141.511395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6141.651141] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6144.977844] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6150.064573] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6150.249962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6155.452948] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6155.652893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6162.429391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6162.432956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6162.449738] LustreError: Skipped 1 previous similar message [ 6162.454862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6162.473449] Lustre: Skipped 6 previous similar messages [ 6162.495372] Lustre: Skipped 3 previous similar messages [ 6164.430414] Lustre: server umount lustre-OST0000 complete [ 6174.690994] Lustre: server umount lustre-MDT0000 complete [ 6177.766885] LustreError: 207616:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6177.792392] LustreError: 207616:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 6178.737913] LustreError: 207602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602235 with bad export cookie 3144371272901507029 [ 6178.766765] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6179.143883] Lustre: server umount lustre-MDT0001 complete [ 6188.671884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6194.359517] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6202.585831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6207.899491] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6216.130483] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6222.027818] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6228.603865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6235.415260] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6243.052632] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6248.076769] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6248.230434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6253.554914] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6253.820815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6257.636060] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6257.640910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6257.659388] Lustre: Skipped 3 previous similar messages [ 6261.978914] Lustre: server umount lustre-OST0000 complete [ 6269.168784] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6274.247851] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6302.919952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6310.250719] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6317.711712] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6321.677990] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6322.999345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.008854] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.016226] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6323.029447] Lustre: Skipped 1 previous similar message [ 6327.804786] Lustre: server umount lustre-OST0000 complete [ 6333.415957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6333.432673] Lustre: Skipped 5 previous similar messages [ 6341.678711] Lustre: server umount lustre-MDT0000 complete [ 6343.651462] LustreError: 211795:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6343.671952] LustreError: 211795:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6344.672354] Lustre: 188616:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789602385/real 1789602385] req@000000007efb338a x1876533312742528/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789602401 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6344.703698] Lustre: 188616:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6345.749243] LustreError: 215904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602402 with bad export cookie 3144371272901507911 [ 6345.759865] LustreError: 215904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6345.761953] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6346.165157] Lustre: server umount lustre-MDT0001 complete [ 6355.059274] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 19:46:51 (1789602411) [ 6361.593300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6362.076213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6362.085943] Lustre: Skipped 6 previous similar messages [ 6366.395902] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6374.751625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6379.786942] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6388.070398] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6394.438720] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6401.400038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6406.913186] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6413.810798] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6418.892806] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6419.135833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6423.772363] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6423.959966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6431.468667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6431.687666] Lustre: lustre-OST0001: new disk, initializing [ 6431.691275] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6431.698755] Lustre: 221132:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6438.013424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6438.026374] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6438.132673] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6438.831804] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6447.791664] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6453.435730] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6453.703947] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6458.579533] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6458.776470] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6463.975365] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6463.984594] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6463.998584] Lustre: Skipped 4 previous similar messages [ 6475.232159] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6475.299496] Lustre: server umount lustre-OST0001 complete [ 6481.568434] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6481.824892] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6486.566518] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6486.798548] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6514.917952] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6520.304313] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6520.317752] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6520.318421] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6520.345430] Lustre: Skipped 1 previous similar message [ 6521.846807] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6530.721200] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6532.480353] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6540.536648] Lustre: server umount lustre-OST0001 complete [ 6572.000128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6572.237257] Lustre: server umount lustre-OST0000 complete [ 6582.606249] Lustre: server umount lustre-MDT0000 complete [ 6586.429148] LustreError: 221131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602643 with bad export cookie 3144371272901508877 [ 6586.433703] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6586.444305] LustreError: 221131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6592.981860] Lustre: server umount lustre-MDT0001 complete [ 6616.941074] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 19:51:13 (1789602673) [ 6623.641382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6623.930685] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6623.985530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6628.801452] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6637.034869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6637.151319] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6637.201892] Lustre: 226382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6642.037948] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6649.997253] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6656.047795] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6663.064392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6663.279567] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6669.413765] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6676.500731] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6685.125635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6685.242990] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6691.530684] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6699.118799] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6700.660776] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6705.201519] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6705.477527] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6709.264445] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6709.502412] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6714.434649] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93c7c54c0800.ost_server_uuid 50 [ 6715.932950] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff93c7c54c0800.ost_server_uuid in IDLE state after 0 sec [ 6716.739699] Lustre: Permanently deactivating lustre-OST0001 [ 6716.758417] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6719.456163] Lustre: Permanently reactivating lustre-OST0001 [ 6724.068670] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6724.078795] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6724.085482] LustreError: Skipped 5 previous similar messages [ 6724.086163] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6724.086167] Lustre: Skipped 21 previous similar messages [ 6724.136676] Lustre: Skipped 7 previous similar messages [ 6729.839993] Lustre: server umount lustre-OST0001 complete [ 6734.310120] LustreError: 228231:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6734.318528] LustreError: 228231:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 40 previous similar messages [ 6739.784405] Lustre: server umount lustre-OST0000 complete [ 6749.858962] Lustre: server umount lustre-MDT0000 complete [ 6753.330513] LustreError: 226377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602810 with bad export cookie 3144371272901510165 [ 6753.333663] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6753.346736] LustreError: 226377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6753.668869] Lustre: server umount lustre-MDT0001 complete [ 6775.566885] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 19:53:51 (1789602831) [ 6783.233873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6783.479168] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6783.511577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6783.518889] Lustre: Skipped 5 previous similar messages [ 6788.333701] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6797.277541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6797.405668] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6797.449839] Lustre: 233075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6801.903880] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6809.385087] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6815.497711] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6822.686544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6822.928435] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6822.964406] Lustre: Permanently deactivating lustre-OST0000 [ 6822.978769] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6828.688848] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6832.694685] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6836.912677] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6844.444095] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6844.587885] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6851.412669] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6852.868989] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6859.793979] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6861.621231] Lustre: Permanently reactivating lustre-OST0000 [ 6861.636489] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6861.639710] Lustre: Skipped 2 previous similar messages [ 6868.094314] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6868.095197] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 36 seconds. I think it's dead, and I am evicting it. exp 00000000a746afb8, cur 1789602925 deadline 1789602919 last 1789602889 [ 6868.107100] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6868.121477] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6868.153800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6883.688154] Lustre: server umount lustre-OST0001 complete [ 6893.871213] Lustre: server umount lustre-OST0000 complete [ 6904.223021] Lustre: server umount lustre-MDT0000 complete [ 6908.363722] LustreError: 233067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602965 with bad export cookie 3144371272901511810 [ 6908.374924] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6908.380495] LustreError: 233067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6908.725320] Lustre: server umount lustre-MDT0001 complete [ 6919.876802] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 6922.617416] Key type lgssc unregistered [ 6922.899864] LNet: 237956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6922.908531] LNetError: 237956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6922.931624] LNet: Removed LNI 192.168.206.159@tcp [ 6923.833717] Key type .llcrypt unregistered [ 6923.838444] Key type ._llcrypt unregistered [ 6947.780800] Key type ._llcrypt registered [ 6947.782682] Key type .llcrypt registered [ 6947.873670] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 6949.049904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6949.153650] alg: No test for adler32 (adler32-zlib) [ 6950.201959] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 6950.529796] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 6952.224212] Key type lgssc registered [ 6953.267874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6962.662712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6962.672504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6963.923518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6967.412709] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6973.430102] LustreError: 239419:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6973.460822] LustreError: 239419:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6977.039775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6977.154567] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6977.187983] Lustre: 239407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6977.196347] Lustre: Permanently deactivating lustre-MDT0001 [ 6977.215691] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6977.413611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6980.227593] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6980.239945] LustreError: 239449:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6980.268625] LustreError: 239449:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6980.321171] LustreError: 239449:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 6983.357060] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 6991.628199] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6997.702807] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7004.392770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7004.776495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7008.882435] LustreError: 241260:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7008.891641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7011.440687] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7014.374720] LustreError: 241260:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7018.531110] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7019.492529] LustreError: 241261:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7022.583115] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7025.543734] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7025.794142] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7031.281812] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7031.286706] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7032.142697] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7039.926841] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7042.755852] Lustre: Permanently reactivating lustre-MDT0001 [ 7042.792322] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7042.797780] Lustre: Skipped 2 previous similar messages [ 7046.725301] Lustre: lustre-MDT0001: Client b15bab1d-7b17-4187-8f17-1a0c3075f1f1 (at 192.168.206.59@tcp) reconnecting [ 7053.652261] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7055.226968] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7061.579586] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7061.821937] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7067.191293] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7067.454885] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7070.012446] Lustre: Permanently deactivating lustre-MDT0001 [ 7070.026454] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7070.031813] Lustre: Skipped 2 previous similar messages [ 7079.004660] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7087.602868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7087.612566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7087.626277] Lustre: Skipped 3 previous similar messages [ 7087.635971] Lustre: Skipped 2 previous similar messages [ 7092.711941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7093.928580] Lustre: server umount lustre-MDT0000 complete [ 7097.825097] LustreError: 240157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603154 with bad export cookie 17205078100892590093 [ 7097.826977] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7097.835534] LustreError: 240157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7097.846710] LustreError: 241279:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7097.846721] LustreError: 241279:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7098.183479] Lustre: server umount lustre-MDT0001 complete [ 7104.140358] Lustre: server umount lustre-OST0000 complete [ 7114.792911] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 19:59:31 (1789603171) [ 7124.574490] Lustre: server umount lustre-OST0001 complete [ 7130.674168] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 7138.586456] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 7174.319633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7174.533984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7174.550327] Lustre: Skipped 2 previous similar messages [ 7174.576495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7174.709820] Lustre: lustre-MDT0000: new disk, initializing [ 7174.769342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7174.798727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7179.233159] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7189.345130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7189.454852] Lustre: 247724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7189.514606] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7189.519561] Lustre: Skipped 1 previous similar message [ 7189.601621] Lustre: lustre-MDT0001: new disk, initializing [ 7189.688694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7189.738094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7189.750898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7193.511865] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7202.569755] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7208.809476] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7215.904302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7216.168473] Lustre: lustre-OST0000: new disk, initializing [ 7216.172213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7216.180769] Lustre: 249704:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7216.258067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7222.611176] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7225.922851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7225.933854] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7226.059032] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7233.262254] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7235.814646] LustreError: 247730:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7245.160569] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7245.447862] Lustre: lustre-OST0001: new disk, initializing [ 7245.451184] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7245.458544] Lustre: 250818:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7245.818430] LustreError: 247730:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7245.878352] LustreError: 248651:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7252.662873] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7253.141870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7253.151854] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7253.261292] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7255.898599] LustreError: 248651:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7263.061701] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7264.736670] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7268.841536] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7268.842501] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7268.854778] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7268.854787] Lustre: Skipped 3 previous similar messages [ 7268.863432] LustreError: Skipped 1 previous similar message [ 7268.914608] Lustre: Skipped 1 previous similar message [ 7273.197899] Lustre: server umount lustre-OST0001 complete [ 7273.957441] LustreError: 249688:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7273.987134] LustreError: 249688:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7292.902484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7292.912378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7292.913962] LustreError: Skipped 1 previous similar message [ 7292.919535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7292.929094] Lustre: Skipped 1 previous similar message [ 7292.955317] Lustre: Skipped 2 previous similar messages [ 7294.444442] LustreError: 249688:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7294.464211] LustreError: 249688:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 7296.599289] Lustre: server umount lustre-OST0000 complete [ 7302.625352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7302.634482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7302.649654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7302.654846] Lustre: Skipped 2 previous similar messages [ 7307.025257] Lustre: server umount lustre-MDT0000 complete [ 7310.174889] LustreError: 247715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603367 with bad export cookie 17205078100892592907 [ 7310.181758] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7310.188223] LustreError: 247715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7316.571265] Lustre: server umount lustre-MDT0001 complete [ 7327.020554] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 7329.854335] Key type lgssc unregistered [ 7330.136436] LNet: 252853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7330.149204] LNetError: 252853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7330.165674] LNet: Removed LNI 192.168.206.159@tcp [ 7331.071848] Key type .llcrypt unregistered [ 7331.074419] Key type ._llcrypt unregistered [ 7369.269205] Key type ._llcrypt registered [ 7369.276212] Key type .llcrypt registered [ 7369.407646] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 7370.426804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7370.530601] alg: No test for adler32 (adler32-zlib) [ 7371.855697] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 7372.145480] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 7373.816292] Key type lgssc registered [ 7375.015413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7382.518829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7382.540199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7383.784761] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7383.811201] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7384.013072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7387.311605] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7394.701733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7394.782947] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7394.811633] Lustre: 255062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7395.018110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7399.139339] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7408.052660] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7415.933263] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7423.767464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7423.893987] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7424.047552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7432.326610] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7433.906100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7441.084870] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7500.769610] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7500.783889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7500.796198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7500.801354] Lustre: Skipped 1 previous similar message [ 7502.425714] Lustre: server umount lustre-OST0000 complete [ 7507.249827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7521.099837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7521.584810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7527.753651] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7536.787046] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7540.995701] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7541.029618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7541.035943] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7582.196184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7582.200252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7582.203534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7582.209262] Lustre: Skipped 2 previous similar messages [ 7582.224055] LustreError: Skipped 1 previous similar message [ 7583.608350] Lustre: server umount lustre-OST0000 complete [ 7587.299152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7587.302450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7587.305474] Lustre: Skipped 1 previous similar message [ 7587.312604] Lustre: Skipped 2 previous similar messages [ 7592.418040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7592.426475] Lustre: Skipped 1 previous similar message [ 7593.405995] Lustre: server umount lustre-MDT0000 complete [ 7597.440995] LustreError: 255054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603654 with bad export cookie 1267762856070281315 [ 7597.449994] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7597.460295] LustreError: 255054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7597.667484] Lustre: server umount lustre-MDT0001 complete [ 7611.263330] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 7614.427982] Key type lgssc unregistered [ 7614.790982] LNet: 260489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7614.803077] LNetError: 260489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7614.830103] LNet: Removed LNI 192.168.206.159@tcp [ 7615.811373] Key type .llcrypt unregistered [ 7615.814869] Key type ._llcrypt unregistered [ 7642.833590] Key type ._llcrypt registered [ 7642.835510] Key type .llcrypt registered [ 7642.949536] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 7643.937780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7644.193163] alg: No test for adler32 (adler32-zlib) [ 7645.245973] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 7645.534414] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 7647.280212] Key type lgssc registered [ 7648.409588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7655.967444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7655.984368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7657.307655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7661.536616] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7667.704048] LustreError: 261905:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7667.722679] LustreError: 261905:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7670.021690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7670.419371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7675.586357] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7683.699676] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7688.989921] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7695.111125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7695.463500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7701.637979] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7702.658192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7709.634279] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7733.734936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7733.736054] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7733.738345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7733.753934] Lustre: Skipped 1 previous similar message [ 7738.862800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7738.869184] Lustre: Skipped 1 previous similar message [ 7739.303901] Lustre: server umount lustre-OST0000 complete [ 7743.971259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7743.976062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7743.994586] Lustre: Skipped 1 previous similar message [ 7744.009535] Lustre: Skipped 2 previous similar messages [ 7749.091919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7749.106263] Lustre: Skipped 1 previous similar message [ 7749.883991] Lustre: server umount lustre-MDT0000 complete [ 7754.022253] LustreError: 261886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603811 with bad export cookie 1750590466408770669 [ 7754.030868] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7754.042104] LustreError: 261886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7754.348756] Lustre: server umount lustre-MDT0001 complete [ 7766.054572] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 7768.826386] Key type lgssc unregistered [ 7769.152366] LNet: 266210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7769.164966] LNetError: 266210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7769.185734] LNet: Removed LNI 192.168.206.159@tcp [ 7770.240440] Key type .llcrypt unregistered [ 7770.242926] Key type ._llcrypt unregistered [ 7788.390042] Key type ._llcrypt registered [ 7788.392835] Key type .llcrypt registered [ 7788.548655] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 7789.831211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7789.865134] alg: No test for adler32 (adler32-zlib) [ 7790.995946] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 7791.237499] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 7792.872194] Key type lgssc registered [ 7794.134841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7803.274424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7803.305981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7804.934896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7809.945796] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7814.120304] LustreError: 267428:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7814.140163] LustreError: 267428:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7819.239205] LustreError: 267429:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7819.738702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7820.217238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7825.410947] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7833.706068] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7840.009157] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7847.083209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7847.541964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7849.606197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7854.800432] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7862.490735] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7880.673590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7880.688750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7880.723254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7881.702422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7881.702756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7881.724085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7883.728681] Lustre: server umount lustre-OST0000 complete [ 7891.940946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7891.944888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7891.972645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7891.979455] Lustre: Skipped 2 previous similar messages [ 7894.764345] Lustre: server umount lustre-MDT0000 complete [ 7897.057044] LustreError: 268411:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7897.079225] LustreError: 268411:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7899.145729] LustreError: 267410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603956 with bad export cookie 14890617024768417580 [ 7899.149273] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7899.157337] LustreError: 267410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7899.584714] Lustre: server umount lustre-MDT0001 complete [ 7912.129955] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 7915.529629] Key type lgssc unregistered [ 7915.901080] LNet: 271107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7915.909637] LNetError: 271107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7915.941324] LNet: Removed LNI 192.168.206.159@tcp [ 7917.045766] Key type .llcrypt unregistered [ 7917.048973] Key type ._llcrypt unregistered [ 7947.309365] Key type ._llcrypt registered [ 7947.313928] Key type .llcrypt registered [ 7947.454165] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 7948.751950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7948.806607] alg: No test for adler32 (adler32-zlib) [ 7949.909707] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 7950.293838] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 7952.112198] Key type lgssc registered [ 7953.320790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7962.541656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7962.568138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7964.081851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7968.826715] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7974.387441] LustreError: 272521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7974.425854] LustreError: 272521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7978.222780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7978.622237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7983.242864] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 7991.208481] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7997.917709] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8006.734946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8007.274884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8009.348348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8014.376442] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8022.556462] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8045.538710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8045.547172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8045.559758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8049.634244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8049.635820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8049.636897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8051.594258] Lustre: server umount lustre-OST0000 complete [ 8055.265251] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8055.273659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8055.287212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8055.292895] Lustre: Skipped 1 previous similar message [ 8059.873775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8059.876152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8059.906162] Lustre: Skipped 1 previous similar message [ 8061.394551] Lustre: server umount lustre-MDT0000 complete [ 8064.994811] LustreError: 272521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8065.030698] LustreError: 272521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8065.478840] LustreError: 272501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604122 with bad export cookie 3717268346603032813 [ 8065.481390] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8065.488671] LustreError: 272501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8065.799831] Lustre: server umount lustre-MDT0001 complete [ 8078.497422] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 8081.763223] Key type lgssc unregistered [ 8082.049626] LNet: 276827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8082.058598] LNetError: 276827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8082.072729] LNet: Removed LNI 192.168.206.159@tcp [ 8082.992745] Key type .llcrypt unregistered [ 8082.994836] Key type ._llcrypt unregistered [ 8100.855582] Key type ._llcrypt registered [ 8100.857982] Key type .llcrypt registered [ 8101.030158] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 8102.129827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8102.284106] alg: No test for adler32 (adler32-zlib) [ 8103.469480] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 8103.793962] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 8105.456436] Key type lgssc registered [ 8106.435986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8113.376224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8113.402384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8115.050536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8121.523827] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8125.417122] LustreError: 278061:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8125.446818] LustreError: 278061:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8130.541830] LustreError: 278063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8132.740276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8133.282530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8138.413297] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8146.732398] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8153.431111] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8161.671590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8162.238786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8169.106437] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8177.385125] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8185.779626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8204.257910] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8204.262186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8204.291168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8206.315588] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8206.329832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8206.330217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8207.833280] Lustre: server umount lustre-OST0000 complete [ 8215.009735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8215.028374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8215.059208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8215.068032] Lustre: Skipped 1 previous similar message [ 8221.667141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8221.673937] Lustre: Skipped 3 previous similar messages [ 8226.272104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8226.492126] Lustre: server umount lustre-MDT0000 complete [ 8226.790917] LustreError: 278060:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8226.804825] LustreError: 278060:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8230.434388] LustreError: 278025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604287 with bad export cookie 15671347348249468468 [ 8230.442992] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8230.450086] LustreError: 278025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8230.815135] Lustre: server umount lustre-MDT0001 complete [ 8242.167844] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 8245.160368] Key type lgssc unregistered [ 8245.593734] LNet: 281785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8245.600761] LNetError: 281785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8245.625314] LNet: Removed LNI 192.168.206.159@tcp [ 8246.707569] Key type .llcrypt unregistered [ 8246.709688] Key type ._llcrypt unregistered [ 8260.244629] Key type ._llcrypt registered [ 8260.246676] Key type .llcrypt registered [ 8260.364427] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8277.570403] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 8293.112955] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 8294.173082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8294.187017] alg: No test for adler32 (adler32-zlib) [ 8295.338104] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 8295.603541] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 8297.360518] Key type lgssc registered [ 8298.853111] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8340.518498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8340.554919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8341.914358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8341.941611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8342.013287] Lustre: lustre-MDT0000: new disk, initializing [ 8342.076720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8342.093657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8347.436254] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8358.092495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8358.196764] Lustre: 285335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8358.270171] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8358.277320] Lustre: Skipped 1 previous similar message [ 8358.420856] Lustre: lustre-MDT0001: new disk, initializing [ 8358.601578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8358.629869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8358.640817] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8364.253722] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8374.476673] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8380.380294] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8386.911821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8387.144177] Lustre: lustre-OST0000: new disk, initializing [ 8387.147349] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8387.153515] Lustre: 287304:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8387.250432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8393.757438] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8395.822832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8395.829059] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8395.939301] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8402.120337] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8407.318784] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8407.638806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8413.462687] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8413.784291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8416.233803] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8416.244120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8416.269982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8416.741882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8421.882059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8421.898932] Lustre: Skipped 2 previous similar messages [ 8422.174362] Lustre: server umount lustre-OST0000 complete [ 8426.977715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8426.996750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8427.005395] Lustre: Skipped 2 previous similar messages [ 8427.017735] Lustre: Skipped 1 previous similar message [ 8432.103354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8432.472178] Lustre: server umount lustre-MDT0000 complete [ 8436.651970] LustreError: 285328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604493 with bad export cookie 14174689038953840991 [ 8436.671734] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8436.677079] LustreError: 285328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8436.964935] Lustre: server umount lustre-MDT0001 complete [ 8446.347725] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 20:21:42 (1789604502) [ 8453.367735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8453.770599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8459.009712] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8459.234502] LustreError: 289717:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8459.267854] LustreError: 289717:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8463.332896] LustreError: 289716:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8468.129324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8468.449598] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8468.460881] Lustre: Skipped 1 previous similar message [ 8468.644438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8473.754911] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8482.324674] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8488.583795] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8495.510983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8495.987795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8502.963683] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8511.396498] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8527.253583] Lustre: server umount lustre-OST0000 complete [ 8533.985955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8533.988123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8533.999553] Lustre: Skipped 1 previous similar message [ 8541.290373] Lustre: server umount lustre-MDT0000 complete [ 8544.228257] Lustre: 283556:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789604585/real 1789604585] req@0000000098a0b610 x1876536173077376/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789604601 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8544.230442] LustreError: 289716:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8544.276342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8544.329963] LustreError: 289716:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8545.881347] LustreError: 289696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604602 with bad export cookie 14174689038953841873 [ 8545.901213] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8545.904887] LustreError: 289696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8546.355802] Lustre: server umount lustre-MDT0001 complete [ 8560.356906] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 8563.301538] Key type lgssc unregistered [ 8563.661748] LNet: 293271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8563.668758] LNetError: 293271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8563.695873] LNet: Removed LNI 192.168.206.159@tcp [ 8564.589097] Key type .llcrypt unregistered [ 8564.593516] Key type ._llcrypt unregistered [ 8595.045901] Key type ._llcrypt registered [ 8595.050731] Key type .llcrypt registered [ 8595.218673] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 8611.590867] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 8613.125576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8613.154035] alg: No test for adler32 (adler32-zlib) [ 8614.325956] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 8614.616636] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 8616.285561] Key type lgssc registered [ 8617.822723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8673.626460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8673.642748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8674.842083] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8674.870790] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8674.948380] Lustre: lustre-MDT0000: new disk, initializing [ 8675.050553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8675.069630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8680.751536] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8693.704568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8693.940611] Lustre: 297094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8694.055656] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8694.062757] Lustre: Skipped 1 previous similar message [ 8694.204467] Lustre: lustre-MDT0001: new disk, initializing [ 8694.377196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8694.439618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8694.453755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8699.313905] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8710.448692] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8716.874136] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8723.999020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8724.329735] Lustre: lustre-OST2710: new disk, initializing [ 8724.333914] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8724.341283] Lustre: 299064:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8724.416806] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8730.209636] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8733.774681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8733.781639] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8733.864086] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8740.553297] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8748.243710] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8748.501264] Lustre: lustre-OST03e8: new disk, initializing [ 8748.508051] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8748.514905] Lustre: 300128:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8748.665541] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8754.290231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8754.301090] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8754.385396] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8754.971628] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8762.595901] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8770.933510] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8771.270792] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8774.935132] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8775.189380] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8781.965470] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8782.248976] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8788.122972] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8788.326166] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8794.897158] Lustre: server umount lustre-MDT0000 complete [ 8795.623249] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8795.651900] Lustre: Skipped 3 previous similar messages [ 8799.238441] LustreError: 297088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604856 with bad export cookie 15296667164147522077 [ 8799.248170] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8799.250604] LustreError: 297088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8799.552470] Lustre: server umount lustre-MDT0001 complete [ 8813.759644] Lustre: server umount lustre-OST2710 complete [ 8816.097907] Lustre: 294892:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789604857/real 1789604857] req@00000000586699bf x1876536507567104/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789604873 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8816.139353] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8817.739271] Lustre: server umount lustre-OST03e8 complete [ 8838.473277] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 8849.180930] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 8887.020852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8887.266533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8887.318735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8887.456659] Lustre: lustre-MDT0000: new disk, initializing [ 8887.544931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8887.570332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8892.567578] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8903.241035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8903.327896] Lustre: 305378:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8903.381178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8903.383765] Lustre: Skipped 1 previous similar message [ 8903.451211] Lustre: lustre-MDT0001: new disk, initializing [ 8903.525279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8903.574234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8903.583857] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8906.900326] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8915.761136] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8921.223775] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8928.602289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8928.857323] Lustre: lustre-OST0000: new disk, initializing [ 8928.862523] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8928.870314] Lustre: 307348:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8928.984276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8934.461508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8934.484385] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8934.524789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8935.211904] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 8945.028124] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8949.893867] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8950.138100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8955.014128] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8955.240086] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8960.484940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8960.499811] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8960.514395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8960.525170] Lustre: Skipped 1 previous similar message [ 8965.601747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8965.610383] Lustre: Skipped 1 previous similar message [ 8970.724158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8970.733821] Lustre: Skipped 1 previous similar message [ 8971.234083] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8971.386681] Lustre: server umount lustre-OST0000 complete [ 8975.332370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8975.340791] LustreError: Skipped 1 previous similar message [ 8975.345593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8975.353130] Lustre: Skipped 1 previous similar message [ 8975.357993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8980.961458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8980.968980] Lustre: Skipped 2 previous similar messages [ 8989.152359] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8989.323732] Lustre: server umount lustre-MDT0000 complete [ 8991.206822] LustreError: 305389:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8991.230403] LustreError: 305389:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 8993.228734] LustreError: 305371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605050 with bad export cookie 15296667164147523764 [ 8993.235243] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8993.245908] LustreError: 305371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8993.545364] Lustre: server umount lustre-MDT0001 complete [ 9002.424115] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 20:30:58 (1789605058) [ 9004.407868] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9006.211379] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 20:31:02 (1789605062) [ 9010.025748] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 9037.089727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9037.536925] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9037.562322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9037.781452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9043.121079] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9051.335179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9051.444856] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9051.470667] Lustre: 311207:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9056.594118] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9078.649793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9078.883784] Lustre: Denying initial registration attempt from nid 192.168.206.159@tcp, specified as failover [ 9078.898972] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9078.909624] LustreError: 312661:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9078.915676] LustreError: 312661:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9078.925574] LustreError: 312661:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9079.020756] Lustre: server umount lustre-OST0000 complete [ 9079.025903] LustreError: 312661:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9087.458310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9087.460949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9087.487528] Lustre: Skipped 1 previous similar message [ 9087.491439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9087.504137] Lustre: Skipped 3 previous similar messages [ 9091.368257] Lustre: server umount lustre-MDT0000 complete [ 9092.586426] LustreError: 311219:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9092.601849] LustreError: 311219:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9095.007373] LustreError: 311201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605152 with bad export cookie 15296667164147524646 [ 9095.008970] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9095.020340] LustreError: 311201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9095.302687] Lustre: server umount lustre-MDT0001 complete [ 9104.502510] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 9107.070265] Key type lgssc unregistered [ 9107.323682] LNet: 313781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9107.330616] LNetError: 313781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9107.344791] LNet: Removed LNI 192.168.206.159@tcp [ 9108.068767] Key type .llcrypt unregistered [ 9108.070558] Key type ._llcrypt unregistered [ 9121.010966] Key type ._llcrypt registered [ 9121.013906] Key type .llcrypt registered [ 9121.121534] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 9122.244860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9122.349131] alg: No test for adler32 (adler32-zlib) [ 9123.469623] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 9123.808159] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 9125.520191] Key type lgssc registered [ 9126.899550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9174.707713] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 9183.085666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9183.106577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9184.392382] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9184.418648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9184.651821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9189.378129] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9198.209734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9198.326357] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9198.366352] Lustre: 316544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9198.569098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9203.214883] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9210.647641] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9216.567826] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9235.241691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9235.476902] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9235.667662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9241.949183] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9249.522174] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9255.394112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9255.407392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9255.423554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9255.907448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9255.921956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9255.954819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9257.712915] Lustre: server umount lustre-OST0000 complete [ 9265.122239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9265.132079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9265.154142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9267.959793] Lustre: server umount lustre-MDT0000 complete [ 9271.580195] LustreError: 316536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605328 with bad export cookie 9919013924747396655 [ 9271.595741] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9271.917777] Lustre: server umount lustre-MDT0001 complete [ 9282.683706] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 9285.115878] Key type lgssc unregistered [ 9285.376771] LNet: 320116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9285.389484] LNetError: 320116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9286.440231] LNet: Removed LNI 192.168.206.159@tcp [ 9287.220559] Key type .llcrypt unregistered [ 9287.223494] Key type ._llcrypt unregistered [ 9313.211549] Key type ._llcrypt registered [ 9313.213367] Key type .llcrypt registered [ 9313.371447] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 9314.526159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9314.542832] alg: No test for adler32 (adler32-zlib) [ 9315.801443] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 9316.236100] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 9317.920286] Key type lgssc registered [ 9319.318862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9326.773925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9326.796472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9328.179150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9332.668797] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9338.344630] LustreError: 321531:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9338.388715] LustreError: 321531:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9340.861403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9341.261111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9345.163383] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9352.212557] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9357.732913] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9366.381504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9366.717834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9372.555970] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9379.909561] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9396.711966] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9396.714208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9396.717917] Lustre: Skipped 1 previous similar message [ 9399.086590] Lustre: server umount lustre-MDT0000 complete [ 9401.828253] LustreError: 322966:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9401.853318] LustreError: 322966:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9403.052698] LustreError: 321512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605460 with bad export cookie 9228234192220982210 [ 9403.064615] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9403.069666] LustreError: 321512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9406.945791] LustreError: 322774:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9406.946651] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9406.970700] LustreError: 322774:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9406.995440] Lustre: Skipped 1 previous similar message [ 9406.998955] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9407.003133] Lustre: Skipped 3 previous similar messages [ 9408.993797] LustreError: 322774:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9408.994670] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9409.012137] LustreError: 322774:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9409.037677] Lustre: Skipped 1 previous similar message [ 9409.454453] Lustre: server umount lustre-MDT0001 complete [ 9414.669731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9423.155727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9428.796522] LustreError: 323364:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9428.891560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9428.972871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9433.815350] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9438.121721] LustreError: 325142:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9438.125711] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9438.136787] LustreError: 325142:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9441.637617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9441.970784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9446.094405] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9447.413573] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9453.260455] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9454.622243] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9459.726348] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9461.394581] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9467.884448] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9467.887407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9467.907283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9467.923172] Lustre: Skipped 1 previous similar message [ 9467.942563] Lustre: Skipped 1 previous similar message [ 9472.998060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9473.011900] Lustre: Skipped 1 previous similar message [ 9480.160117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9480.340722] Lustre: server umount lustre-OST0000 complete [ 9488.353661] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9488.356134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9488.358297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9488.358305] Lustre: Skipped 2 previous similar messages [ 9488.380600] Lustre: Skipped 1 previous similar message [ 9490.008829] Lustre: server umount lustre-MDT0000 complete [ 9493.411909] LustreError: 321511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605550 with bad export cookie 9228234192220989343 [ 9493.419877] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9493.425281] LustreError: 321511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9493.723630] Lustre: server umount lustre-MDT0001 complete [ 9505.416286] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 9508.211414] Key type lgssc unregistered [ 9508.537073] LNet: 327817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9508.551932] LNetError: 327817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9508.586643] LNet: Removed LNI 192.168.206.159@tcp [ 9509.574302] Key type .llcrypt unregistered [ 9509.576209] Key type ._llcrypt unregistered [ 9535.459923] Key type ._llcrypt registered [ 9535.464437] Key type .llcrypt registered [ 9535.571336] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 9536.836382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9536.886976] alg: No test for adler32 (adler32-zlib) [ 9538.056899] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 9538.417540] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 9540.104200] Key type lgssc registered [ 9541.387594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9549.446422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9549.469518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9550.918225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9554.888253] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9560.044226] LustreError: 329230:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9560.064957] LustreError: 329230:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9562.070545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9562.320340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9566.674744] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9574.123745] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9579.301909] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9583.074351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9583.085320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9583.104315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9586.147046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9586.148142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9587.173138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9587.187690] Lustre: Skipped 2 previous similar messages [ 9588.731100] Lustre: server umount lustre-MDT0000 complete [ 9592.290899] LustreError: 329231:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9592.314767] LustreError: 329231:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9592.417853] LustreError: 329213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605649 with bad export cookie 15294727484270941833 [ 9592.441790] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9592.449617] LustreError: 329213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9592.765615] Lustre: server umount lustre-MDT0001 complete [ 9608.000534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9608.404811] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9608.441885] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9608.746798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9613.184715] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9622.296938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9622.413039] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9622.441984] Lustre: 331816:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9622.654252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9626.823298] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9634.296848] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9640.280500] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9648.622784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9648.858344] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9648.866906] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9648.872739] LustreError: 333701:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 9648.876758] LustreError: 333701:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9648.880692] LustreError: 333701:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9648.940443] Lustre: server umount lustre-OST0000 complete [ 9648.946898] LustreError: 333701:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9659.256920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9659.588744] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9659.807231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9665.932207] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9669.781983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9673.598371] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9681.622057] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9681.755329] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9681.819818] Lustre: lustre-OST0001: new disk, initializing [ 9681.823712] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9681.829580] Lustre: 335243:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9681.910403] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9688.858633] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9690.745951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9690.785569] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9690.893992] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9696.753511] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9701.350839] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9701.364254] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9701.389614] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9701.417167] Lustre: Skipped 1 previous similar message [ 9706.231702] Lustre: server umount lustre-OST0001 complete [ 9706.472434] LustreError: 334167:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9706.486163] LustreError: 334167:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9710.564854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9710.585216] LustreError: Skipped 1 previous similar message [ 9710.593201] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9710.606587] Lustre: Skipped 1 previous similar message [ 9710.615802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9711.592985] LustreError: 334170:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9711.619598] LustreError: 334170:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9715.622639] Lustre: server umount lustre-OST0000 complete [ 9719.780925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9719.791656] LustreError: Skipped 1 previous similar message [ 9719.797090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9719.812739] Lustre: Skipped 1 previous similar message [ 9719.819359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9719.826895] Lustre: Skipped 2 previous similar messages [ 9725.262530] Lustre: server umount lustre-MDT0000 complete [ 9726.955143] LustreError: 331822:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9726.973158] LustreError: 331822:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9728.905688] LustreError: 335242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605785 with bad export cookie 15294727484270942386 [ 9728.916482] LustreError: 335242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9728.918168] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9729.199572] Lustre: server umount lustre-MDT0001 complete [ 9738.278745] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [ 9740.577814] Key type lgssc unregistered [ 9740.799777] LNet: 337200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9740.811177] LNetError: 337200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9740.829377] LNet: Removed LNI 192.168.206.159@tcp [ 9741.538242] Key type .llcrypt unregistered [ 9741.542811] Key type ._llcrypt unregistered [ 9812.469918] Key type ._llcrypt registered [ 9812.478793] Key type .llcrypt registered [ 9812.725799] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [ 9828.849197] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [ 9830.020391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9830.039629] alg: No test for adler32 (adler32-zlib) [ 9831.065463] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 9831.265198] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [ 9832.904188] Key type lgssc registered [ 9834.072055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9866.082871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9866.107878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9867.397456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9867.437929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9867.536460] Lustre: lustre-MDT0000: new disk, initializing [ 9867.613482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9867.630994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9872.032215] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9882.574197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9882.754121] Lustre: 342129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9882.799260] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9882.804313] Lustre: Skipped 1 previous similar message [ 9882.871681] Lustre: lustre-MDT0001: new disk, initializing [ 9882.955478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9883.005735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9883.016315] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9886.607634] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9895.386205] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9900.829201] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9906.338894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9906.582951] Lustre: lustre-OST0000: new disk, initializing [ 9906.588195] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9906.594112] Lustre: 344098:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9906.671240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9911.707355] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9911.858762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9911.867958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9911.966073] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9921.378349] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9925.381529] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9925.620528] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9930.394291] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9930.649294] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9932.774424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9932.798859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9932.801370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9937.381338] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9937.393114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9937.417692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9938.762669] Lustre: server umount lustre-OST0000 complete [ 9943.011617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9943.031050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9943.039393] Lustre: Skipped 2 previous similar messages [ 9947.827987] Lustre: server umount lustre-MDT0000 complete [ 9951.055291] LustreError: 342122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789606008 with bad export cookie 5531004814567552079 [ 9951.070284] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9951.075053] LustreError: 342122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9951.330386] Lustre: server umount lustre-MDT0001 complete [ 9959.454600] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 20:46:56 (1789606016) [ 9965.364466] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 20:47:02 (1789606022) [ 9972.220036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9972.552575] LustreError: 346854:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9972.566799] LustreError: 346854:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9972.625344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9977.538990] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9977.839303] LustreError: 346855:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9982.953079] LustreError: 346854:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9986.987180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9987.561908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9991.903394] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [ 9998.187878] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10004.238164] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10011.907320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10012.137310] LustreError: 348674:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10012.286079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10017.918195] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10025.204634] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10032.270398] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10033.983194] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10035.472959] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10037.110046] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10039.079142] Lustre: Failing over lustre-MDT0000 [10039.365252] Lustre: server umount lustre-MDT0000 complete [10042.628270] LustreError: 346851:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10043.363064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10043.373848] Lustre: Skipped 3 previous similar messages [10052.817090] LustreError: 349297:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10052.832762] LustreError: 349297:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [10058.009667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10058.159936] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10058.439118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10058.493314] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10059.010468] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10063.012946] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10063.851285] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10063.906821] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10063.959849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10071.842610] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10073.461874] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10076.984234] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10079.056640] Lustre: Failing over lustre-MDT0000 [10079.200770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10079.203878] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10079.219467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10079.231340] Lustre: Skipped 1 previous similar message [10079.247816] Lustre: Skipped 2 previous similar messages [10079.312243] Lustre: server umount lustre-MDT0000 complete [10079.486179] LustreError: 346851:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10079.506016] LustreError: 346851:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10098.510748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10098.614940] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10098.911572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10098.986903] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10101.463934] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10103.355382] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10104.305076] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10104.310415] Lustre: Skipped 2 previous similar messages [10104.357513] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10104.396142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10112.527064] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10114.397365] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10124.781187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10124.782443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10124.789625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10124.811463] Lustre: Skipped 2 previous similar messages [10126.243621] Lustre: server umount lustre-MDT0000 complete [10129.892503] LustreError: 346851:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10129.927857] LustreError: 346851:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [10129.964989] LustreError: 348698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789606186 with bad export cookie 5531004814567554788 [10129.968069] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10129.973397] LustreError: 348698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10130.581055] Lustre: server umount lustre-MDT0001 complete [10144.901815] Lustre: server umount lustre-OST0000 complete [10157.880453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10161.489340] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10168.751487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10169.123126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10169.130991] Lustre: Skipped 1 previous similar message [10173.475943] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10180.045552] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10185.630619] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10191.952965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10198.216413] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10203.580284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10205.738076] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10213.867574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10213.876282] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10213.876473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10213.876478] Lustre: Skipped 2 previous similar messages [10213.879465] Lustre: Skipped 2 previous similar messages [10216.788558] Lustre: server umount lustre-OST0000 complete [10226.786564] Lustre: server umount lustre-MDT0000 complete [10229.221884] LustreError: 353196:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10229.257815] LustreError: 353196:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10230.529463] LustreError: 353177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789606287 with bad export cookie 5531004814567555537 [10230.539835] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10230.917231] Lustre: server umount lustre-MDT0001 complete [10243.279955] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [10246.172279] Key type lgssc unregistered [10246.505077] LNet: 356725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10246.516226] LNetError: 356725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10247.591266] LNet: Removed LNI 192.168.206.159@tcp [10248.897241] Key type .llcrypt unregistered [10248.901653] Key type ._llcrypt unregistered [10274.606633] Key type ._llcrypt registered [10274.608054] Key type .llcrypt registered [10274.724906] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [10275.578238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10275.596471] alg: No test for adler32 (adler32-zlib) [10276.603947] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [10276.774295] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [10278.424265] Key type lgssc registered [10279.578699] Lustre: Echo OBD driver; http://www.lustre.org/ [10289.666638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10289.720575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10291.876180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10296.610063] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10301.424866] LustreError: 358237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10301.454093] LustreError: 358237:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10306.532394] LustreError: 358236:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10307.453133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10307.752554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10312.480481] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10319.841321] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10325.797647] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10332.511266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10333.025880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10336.114941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10339.370192] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10346.748201] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10364.403954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10364.417350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10364.444897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10367.972103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10367.972272] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10367.988304] Lustre: Skipped 1 previous similar message [10368.003847] Lustre: Skipped 1 previous similar message [10369.853557] Lustre: server umount lustre-MDT0000 complete [10373.091742] LustreError: 358965:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10373.131484] LustreError: 358965:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10374.759858] LustreError: 358218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789606431 with bad export cookie 2712565768750253455 [10374.766511] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10374.779607] LustreError: 358218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10375.085851] Lustre: server umount lustre-MDT0001 complete [10389.514488] Lustre: server umount lustre-OST0000 complete [10404.151602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10404.887361] LustreError: 361970:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10405.006464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10410.468439] LustreError: 361971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10410.745038] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10419.684961] LustreError: 361971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10419.717025] LustreError: 361971:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10419.938201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10420.309389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10425.288550] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10433.350291] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10439.582844] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10446.951559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10447.364907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10449.409946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10454.223753] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10462.548447] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10476.008730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10476.013487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10476.035813] Lustre: Skipped 2 previous similar messages [10476.043584] Lustre: Skipped 1 previous similar message [10480.052797] Lustre: server umount lustre-MDT0000 complete [10481.128866] LustreError: 361971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10481.149570] LustreError: 361971:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10483.487765] LustreError: 361952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789606540 with bad export cookie 2712565768750254561 [10483.499655] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10483.511587] LustreError: 361952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10483.739151] Lustre: server umount lustre-MDT0001 complete [10498.138489] Lustre: server umount lustre-OST0000 complete [10512.984525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10513.278158] LustreError: 365883:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10513.329839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10517.897922] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10528.270338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10532.847531] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10542.351777] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10549.619697] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10556.246939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10556.636965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10556.647553] Lustre: Skipped 1 previous similar message [10558.714283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10563.336236] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10571.506703] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10578.408764] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10578.422338] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10578.442986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10578.465929] Lustre: Skipped 2 previous similar messages [10582.822304] Lustre: server umount lustre-OST0000 complete [10588.642843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10588.674810] Lustre: Skipped 1 previous similar message [10588.692458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10592.980501] Lustre: server umount lustre-MDT0000 complete [10593.762128] LustreError: 365880:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10593.780964] LustreError: 365880:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10597.268706] LustreError: 365864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789606654 with bad export cookie 2712565768750255590 [10597.284608] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10597.690023] Lustre: server umount lustre-MDT0001 complete [10608.894431] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [10611.740213] Key type lgssc unregistered [10612.072117] LNet: 369413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10612.084759] LNetError: 369413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10613.161908] LNet: Removed LNI 192.168.206.159@tcp [10614.248414] Key type .llcrypt unregistered [10614.254220] Key type ._llcrypt unregistered [10640.725764] Key type ._llcrypt registered [10640.739046] Key type .llcrypt registered [10640.846897] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [10642.536712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10642.619381] alg: No test for adler32 (adler32-zlib) [10643.788829] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [10644.088993] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [10645.808256] Key type lgssc registered [10647.133456] Lustre: Echo OBD driver; http://www.lustre.org/ [10654.983231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10655.019383] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10655.031608] LustreError: 370852:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10655.085694] LustreError: 370852:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10655.093940] LustreError: 370852:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10655.101951] LustreError: 370852:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10655.115900] LustreError: 370852:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10664.395102] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10664.406229] LustreError: 371249:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10664.425951] LustreError: 371249:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10664.431571] LustreError: 371249:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10664.437632] LustreError: 371249:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10664.470700] LustreError: 371249:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10687.742293] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [10691.137252] Key type lgssc unregistered [10691.524575] LNet: 372253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10691.534276] LNetError: 372253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10691.562256] LNet: Removed LNI 192.168.206.159@tcp [10692.466765] Key type .llcrypt unregistered [10692.468744] Key type ._llcrypt unregistered [10711.560244] Key type ._llcrypt registered [10711.564521] Key type .llcrypt registered [10711.692707] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [10728.548964] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [10729.250298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10729.360106] alg: No test for adler32 (adler32-zlib) [10730.429255] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [10730.662935] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [10732.390487] Key type lgssc registered [10733.934264] Lustre: Echo OBD driver; http://www.lustre.org/ [10777.868777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10777.915072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10779.187689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10779.224249] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10779.327197] Lustre: lustre-MDT0000: new disk, initializing [10779.468414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10779.494883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10783.767878] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10794.751264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10794.912953] Lustre: 375407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10794.978711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10794.984527] Lustre: Skipped 1 previous similar message [10795.084947] Lustre: lustre-MDT0001: new disk, initializing [10795.195831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10795.238206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10795.250452] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10799.749361] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10809.523755] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10816.183065] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10823.058275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10823.398227] Lustre: lustre-OST0000: new disk, initializing [10823.401276] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10823.407825] Lustre: 377388:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10823.473212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10829.435130] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10831.397363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10831.406419] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10831.536704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10839.357991] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10844.422612] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10844.679812] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10848.397928] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10848.623363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10851.812558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10851.821380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10851.838959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10851.842621] Lustre: Skipped 1 previous similar message [10856.435576] Lustre: server umount lustre-OST0000 complete [10862.050507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10862.055658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10862.070354] Lustre: Skipped 2 previous similar messages [10862.088803] Lustre: Skipped 1 previous similar message [10868.754320] Lustre: server umount lustre-MDT0000 complete [10872.171094] LustreError: 375401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789606929 with bad export cookie 10354315824203547952 [10872.185676] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10872.195429] LustreError: 375401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10872.596846] Lustre: server umount lustre-MDT0001 complete [10882.269261] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10883.886375] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 21:02:20 (1789606940) [10891.902064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10892.680973] LustreError: 379848:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10892.701683] LustreError: 379848:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10892.761091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10896.979449] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10897.891154] LustreError: 379849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10901.993487] LustreError: 379848:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10904.961326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10905.350953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10909.001801] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10915.823680] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10920.829714] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10926.647437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10926.902465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10932.148665] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10939.095085] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10945.358264] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10945.518950] Lustre: lustre-OST0001: new disk, initializing [10945.522577] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10945.528716] Lustre: 382688:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10945.663365] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10953.387891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10953.401286] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10953.594336] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10953.907504] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [10963.875148] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10969.061228] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10969.063668] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10969.065198] LustreError: Skipped 1 previous similar message [10969.066068] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10969.070816] Lustre: Skipped 1 previous similar message [10969.080407] Lustre: Skipped 1 previous similar message [10971.898370] Lustre: lustre-OST0001: Not available for connect from 192.168.206.59@tcp (stopping) [10974.582341] Lustre: server umount lustre-OST0001 complete [10976.974860] LustreError: 381668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10984.443100] LustreError: 381671:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10984.483621] LustreError: 381671:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10992.610746] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10992.623252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10992.649386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10992.661212] Lustre: Skipped 2 previous similar messages [10994.662733] LustreError: 381670:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10994.728255] LustreError: 381670:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10997.117928] Lustre: server umount lustre-OST0000 complete [11002.849413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11002.859036] LustreError: Skipped 1 previous similar message [11002.862988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11002.876634] Lustre: Skipped 1 previous similar message [11002.880101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11002.886336] Lustre: Skipped 3 previous similar messages [11007.440903] Lustre: server umount lustre-MDT0000 complete [11011.408598] LustreError: 379829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607068 with bad export cookie 10354315824203548834 [11011.415696] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11011.419302] LustreError: 379829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11017.875981] Lustre: server umount lustre-MDT0001 complete [11028.874395] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [11031.567475] Key type lgssc unregistered [11031.882684] LNet: 384675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11031.894600] LNetError: 384675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11031.924821] LNet: Removed LNI 192.168.206.159@tcp [11032.877605] Key type .llcrypt unregistered [11032.879387] Key type ._llcrypt unregistered [11082.737924] Key type ._llcrypt registered [11082.740550] Key type .llcrypt registered [11082.807327] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [11083.601142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11083.631655] alg: No test for adler32 (adler32-zlib) [11084.657812] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [11084.886374] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [11086.552601] Key type lgssc registered [11087.434536] Lustre: Echo OBD driver; http://www.lustre.org/ [11096.345841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11096.400719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11097.846767] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11097.866816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11098.016361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11103.532082] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11111.655406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11111.785301] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11111.813607] Lustre: 387469:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11112.065375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11117.026720] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11124.823839] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11130.191405] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11132.897150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11132.921481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11137.505471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11137.522841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11137.545481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11138.806354] Lustre: server umount lustre-MDT0000 complete [11143.002859] LustreError: 387460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607200 with bad export cookie 9139145066371827756 [11143.014786] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11143.023554] LustreError: 387460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11143.349265] Lustre: server umount lustre-MDT0001 complete [11154.584939] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 21:06:50 (1789607210) [11160.612257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11161.076338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11165.221869] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11166.181587] LustreError: 389982:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11166.197462] LustreError: 389982:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11171.311840] LustreError: 389981:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11172.602916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11172.921255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11176.618177] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11183.330189] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11189.272281] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11196.221544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11196.448399] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11196.605961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11202.595437] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11208.933682] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11215.060488] Lustre: Permanently deactivating lustre-OST0000 [11215.078866] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11221.503810] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11223.901819] LustreError: 392969:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11223.919565] LustreError: 392969:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11223.927304] LustreError: 392969:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11226.118876] LustreError: 393018:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11226.125703] LustreError: 393018:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11234.028943] Lustre: server umount lustre-OST0000 complete [11236.248146] LustreError: 393267:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11236.254040] LustreError: 393267:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11239.403572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11239.423323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11239.446571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11239.458555] Lustre: Skipped 2 previous similar messages [11246.049429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11246.060843] Lustre: Skipped 3 previous similar messages [11248.233245] Lustre: server umount lustre-MDT0000 complete [11251.170171] LustreError: 389976:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11251.192644] LustreError: 389976:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11251.595324] LustreError: 390712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607308 with bad export cookie 9139145066371828337 [11251.602491] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11251.608775] LustreError: 390712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11262.029581] Lustre: server umount lustre-MDT0001 complete [11269.766171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11273.738964] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11276.134426] LustreError: 394426:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11281.907576] Lustre: 394764:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.159@tcp [11283.713066] Lustre: server umount MGS complete [11294.338505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11294.650157] LustreError: 395489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11294.657514] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11294.702465] Lustre: Skipped 1 previous similar message [11294.765581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11299.270480] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11299.812527] LustreError: 395490:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11306.928333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11307.216213] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11311.808398] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11320.151647] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11326.806774] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11328.811368] Lustre: Permanently reactivating lustre-OST0000 [11328.823335] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11328.830613] Lustre: Skipped 2 previous similar messages [11356.130518] LustreError: 397271:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11364.946668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11365.382279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11365.391831] Lustre: Skipped 1 previous similar message [11371.734734] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11378.805287] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11385.826678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11385.840224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11385.863152] Lustre: Skipped 1 previous similar message [11385.867648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11386.298693] Lustre: server umount lustre-OST0000 complete [11391.976650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11391.997555] Lustre: Skipped 1 previous similar message [11397.090499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11397.098487] Lustre: Skipped 3 previous similar messages [11400.314149] Lustre: server umount lustre-MDT0000 complete [11401.190892] LustreError: 396508:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11401.207124] LustreError: 396508:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11403.866343] LustreError: 395469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607460 with bad export cookie 9139145066371829387 [11403.872326] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11403.872524] LustreError: 395469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11404.173655] Lustre: server umount lustre-MDT0001 complete [11416.266476] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [11419.301486] Key type lgssc unregistered [11419.548332] LNet: 399856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11419.567791] LNetError: 399856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11420.653564] LNet: Removed LNI 192.168.206.159@tcp [11421.561627] Key type .llcrypt unregistered [11421.563689] Key type ._llcrypt unregistered [11439.460441] Key type ._llcrypt registered [11439.462452] Key type .llcrypt registered [11439.547811] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [11452.847763] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [11453.666353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11453.738666] alg: No test for adler32 (adler32-zlib) [11454.837392] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [11455.039557] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [11456.731526] Key type lgssc registered [11458.034669] Lustre: Echo OBD driver; http://www.lustre.org/ [11495.558206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11495.604021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11496.948738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11496.996960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11497.101947] Lustre: lustre-MDT0000: new disk, initializing [11497.201572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11497.240910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11501.759717] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11511.759535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11511.876221] Lustre: 403003:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11511.933138] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11511.941613] Lustre: Skipped 1 previous similar message [11512.009140] Lustre: lustre-MDT0001: new disk, initializing [11512.141160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11512.180255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11512.198736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11517.076887] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11526.989991] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11532.391529] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11540.087698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11540.341480] Lustre: lustre-OST0000: new disk, initializing [11540.345683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11540.355535] Lustre: 404972:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11540.428403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11545.666714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11545.680862] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11545.816450] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11547.062762] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11556.463958] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11560.875255] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11561.092474] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11564.793143] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11564.938420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11566.567341] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11566.573657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11566.578170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11566.586985] Lustre: Skipped 1 previous similar message [11571.694232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11571.698749] Lustre: Skipped 1 previous similar message [11572.635335] Lustre: server umount lustre-OST0000 complete [11576.808559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11576.819338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11576.824627] Lustre: Skipped 1 previous similar message [11576.840467] Lustre: Skipped 2 previous similar messages [11581.747860] Lustre: server umount lustre-MDT0000 complete [11584.921248] LustreError: 402994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607641 with bad export cookie 2627995898514892057 [11584.937345] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11585.209504] Lustre: server umount lustre-MDT0001 complete [11592.618469] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11594.286408] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 21:14:10 (1789607650) [11600.119471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11600.496844] LustreError: 407430:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11600.518064] LustreError: 407430:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11600.567863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11603.919678] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11605.988111] LustreError: 407431:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11610.085868] LustreError: 407430:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11610.669244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11610.997844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11614.762331] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11621.901108] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11627.656639] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11634.193193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11634.542907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11641.112966] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11647.684754] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11649.187593] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11651.697884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11661.283303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11661.288480] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11661.295146] LustreError: Skipped 1 previous similar message [11661.306802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11663.306086] Lustre: server umount lustre-OST0000 complete [11667.431831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11667.440388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11667.460841] Lustre: Skipped 1 previous similar message [11667.465265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11667.469983] Lustre: Skipped 1 previous similar message [11671.523837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11673.512423] Lustre: server umount lustre-MDT0000 complete [11676.641298] LustreError: 408494:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11676.663606] LustreError: 408494:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11677.645196] LustreError: 407412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607734 with bad export cookie 2627995898514892939 [11677.653058] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11677.656616] LustreError: 407412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11677.967843] Lustre: server umount lustre-MDT0001 complete [11687.339390] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [11689.440540] Key type lgssc unregistered [11689.616196] LNet: 411155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11689.622791] LNetError: 411155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11689.646592] LNet: Removed LNI 192.168.206.159@tcp [11690.187506] Key type .llcrypt unregistered [11690.189588] Key type ._llcrypt unregistered [11711.632727] Key type ._llcrypt registered [11711.635268] Key type .llcrypt registered [11711.730706] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [11711.988473] Key type .llcrypt unregistered [11711.993323] Key type ._llcrypt unregistered [11728.140446] Key type ._llcrypt registered [11728.142901] Key type .llcrypt registered [11728.253511] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [11729.278223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11729.428425] alg: No test for adler32 (adler32-zlib) [11730.501334] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [11730.775739] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [11732.456314] Key type lgssc registered [11733.547668] Lustre: Echo OBD driver; http://www.lustre.org/ [11739.893955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11739.911230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11741.192782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11745.403309] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11751.394762] LustreError: 413358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11751.423379] LustreError: 413358:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11753.551979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11753.852375] LustreError: 413357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11753.874463] LustreError: 413357:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11753.888080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11759.076664] LustreError: 413358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11759.103653] LustreError: 413358:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11759.538516] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11764.199412] LustreError: 413357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11764.216597] LustreError: 413357:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11766.259987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11766.854718] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11771.362868] LustreError: 413358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11771.379340] LustreError: 413358:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11773.897904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11774.254896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11778.254375] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11788.980940] Lustre: server umount lustre-OST0000 complete [11794.916637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11794.916904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11794.918645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11794.949163] Lustre: Skipped 1 previous similar message [11800.035825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11800.045349] Lustre: Skipped 2 previous similar messages [11802.174444] Lustre: server umount lustre-MDT0000 complete [11805.152461] Lustre: 412667:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789607846/real 1789607846] req@000000003d981880 x1876539774887936/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789607862 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11805.156870] LustreError: 413357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11805.188109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11805.230991] LustreError: 413357:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11805.805095] LustreError: 413337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607862 with bad export cookie 6466313478616282189 [11805.825544] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11806.169216] Lustre: server umount lustre-MDT0001 complete [11816.857971] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [11820.018419] Key type lgssc unregistered [11820.437544] LNet: 416503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11820.449695] LNetError: 416503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11820.462653] LNet: Removed LNI 192.168.206.159@tcp [11821.440974] Key type .llcrypt unregistered [11821.443799] Key type ._llcrypt unregistered [11845.795505] Key type ._llcrypt registered [11845.798848] Key type .llcrypt registered [11845.925901] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [11846.877139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11846.969203] alg: No test for adler32 (adler32-zlib) [11847.955243] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [11848.184514] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [11849.856584] Key type lgssc registered [11850.995911] Lustre: Echo OBD driver; http://www.lustre.org/ [11858.618463] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11858.648716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11874.208398] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11879.328354] LustreError: 417888:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.159@tcp: failed processing log, type 4: rc = -110 [11905.952180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11912.297087] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11919.092714] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11925.161895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11925.608424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11925.670306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11929.800433] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11931.110467] LustreError: 417913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11931.125873] LustreError: 417913:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11936.226130] LustreError: 417914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11937.577873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11937.979886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11942.345777] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [11950.086328] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11956.059342] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11963.874401] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11963.887654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11963.905482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11965.415386] Lustre: server umount lustre-OST0000 complete [11968.994939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11969.010061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11969.031965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11971.046316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11971.046336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11971.076214] Lustre: Skipped 1 previous similar message [11976.167851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11976.177066] Lustre: Skipped 1 previous similar message [11978.282123] Lustre: server umount lustre-MDT0000 complete [11980.261414] LustreError: 418930:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11980.290870] LustreError: 418930:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11981.602903] LustreError: 417894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789608038 with bad export cookie 17135083391843753887 [11981.606397] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11981.613909] LustreError: 417894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11981.878551] Lustre: server umount lustre-MDT0001 complete [11993.337514] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [11996.423193] Key type lgssc unregistered [11996.714668] LNet: 421458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11996.739924] LNetError: 421458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11996.777952] LNet: Removed LNI 192.168.206.159@tcp [11998.075222] Key type .llcrypt unregistered [11998.078516] Key type ._llcrypt unregistered [12023.836581] Key type ._llcrypt registered [12023.840305] Key type .llcrypt registered [12023.976144] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [12025.296373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12025.317371] alg: No test for adler32 (adler32-zlib) [12026.396466] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [12026.647411] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [12028.304593] Key type lgssc registered [12029.609306] Lustre: Echo OBD driver; http://www.lustre.org/ [12037.674650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12037.690621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12039.151633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12043.735394] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [12049.387627] LustreError: 422871:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12049.428163] LustreError: 422871:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12052.670362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12053.019572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12056.155970] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [12062.372544] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12069.303671] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12076.388644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12076.726876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12082.976795] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [12084.929311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12090.743852] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12095.461033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12095.479614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12095.489767] Lustre: Skipped 1 previous similar message [12097.772139] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.59@tcp (stopping) [12100.584779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12100.593501] Lustre: Skipped 2 previous similar messages [12101.234387] Lustre: server umount lustre-MDT0000 complete [12102.869530] LustreError: 422868:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12103.611164] LustreError: 422851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.59@tcp arrived at 1789608160 with bad export cookie 2671038793699312250 [12103.624350] LustreError: 422851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12105.704864] LustreError: 422871:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12105.737405] LustreError: 422871:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12105.843403] LustreError: 422853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789608162 with bad export cookie 2671038793699311354 [12105.869653] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12108.770094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12108.776923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12108.789766] Lustre: Skipped 2 previous similar messages [12108.795656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12110.817992] LustreError: 422867:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12110.841579] LustreError: 422867:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12112.300173] Lustre: server umount lustre-OST0000 complete [12115.946342] LustreError: 422866:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12115.970881] LustreError: 422866:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12119.944921] Lustre: server umount lustre-MDT0001 complete [12131.407321] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [12134.382819] Key type lgssc unregistered [12134.753777] LNet: 426554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12134.762325] LNetError: 426554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12134.776207] LNet: Removed LNI 192.168.206.159@tcp [12135.937845] Key type .llcrypt unregistered [12135.940236] Key type ._llcrypt unregistered [12165.951802] Key type ._llcrypt registered [12165.955394] Key type .llcrypt registered [12166.079795] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [12167.075875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12167.170312] alg: No test for adler32 (adler32-zlib) [12168.231233] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [12168.414726] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [12170.040253] Key type lgssc registered [12171.104775] Lustre: Echo OBD driver; http://www.lustre.org/ [12178.287695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12178.307210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12179.753474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12184.012224] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [12189.157485] LustreError: 427973:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12189.189902] LustreError: 427973:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12193.037074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12193.404344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12197.438336] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [12204.884628] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12210.812863] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12217.081951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12217.460769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12223.611938] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [12224.623918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12230.597961] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12234.721333] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12234.732898] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12234.758881] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12235.242329] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12235.261753] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12235.268842] Lustre: Skipped 1 previous similar message [12237.573675] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.59@tcp (stopping) [12240.304888] Lustre: server umount lustre-MDT0001 complete [12240.356052] LustreError: 428702:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12240.375272] LustreError: 428702:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12245.474400] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12245.476316] LustreError: 427974:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12245.485846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12245.488070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12250.595688] LustreError: 427973:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12250.597558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12251.072923] Lustre: server umount lustre-OST0000 complete [12254.497629] Lustre: server umount lustre-MDT0000 complete [12265.250722] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [12267.590632] Key type lgssc unregistered [12267.824782] LNet: 431658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12267.830369] LNetError: 431658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12268.903305] LNet: Removed LNI 192.168.206.159@tcp [12269.769752] Key type .llcrypt unregistered [12269.771501] Key type ._llcrypt unregistered [12295.189129] Key type ._llcrypt registered [12295.190607] Key type .llcrypt registered [12295.273732] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [12310.375197] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [12311.416583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12311.431757] alg: No test for adler32 (adler32-zlib) [12312.533919] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [12312.789155] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [12314.440464] Key type lgssc registered [12315.682887] Lustre: Echo OBD driver; http://www.lustre.org/ [12350.766400] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [12365.649945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12365.686197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12366.889702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12366.921155] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12366.994774] Lustre: lustre-MDT0000: new disk, initializing [12367.051321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12367.064618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12370.978358] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [12385.556832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12385.892214] Lustre: lustre-OST0000: new disk, initializing [12385.896322] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12385.900122] Lustre: Skipped 1 previous similar message [12385.907221] Lustre: 436686:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12385.984302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12393.133435] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [12394.107133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12394.120812] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12394.174342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12402.844769] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12420.541706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12420.678670] Lustre: 435667:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12420.750669] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12420.823388] Lustre: lustre-MDT0001: new disk, initializing [12421.037261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12421.097594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12421.111139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12427.178869] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [12440.018319] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12440.300618] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12445.942455] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12446.247360] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12454.372989] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12454.398578] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12454.408819] Lustre: Skipped 1 previous similar message [12455.673136] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.59@tcp (stopping) [12457.480731] Lustre: server umount lustre-MDT0001 complete [12464.610243] LustreError: 438020:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12464.640937] LustreError: 438020:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12469.739066] LustreError: 435674:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12469.763504] LustreError: 435674:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12474.856782] LustreError: 435680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12474.861322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12474.886618] Lustre: Skipped 1 previous similar message [12474.893159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12479.972834] LustreError: 438012:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12479.972912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12480.565809] Lustre: server umount lustre-OST0000 complete [12490.138674] Lustre: server umount lustre-MDT0000 complete [12502.336940] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [12504.831513] Key type lgssc unregistered [12505.055659] LNet: 440434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12505.059674] LNetError: 440434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12506.086968] LNet: Removed LNI 192.168.206.159@tcp [12506.882511] Key type .llcrypt unregistered [12506.886351] Key type ._llcrypt unregistered [12544.813912] Key type ._llcrypt registered [12544.817510] Key type .llcrypt registered [12544.910986] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [12545.154478] Key type .llcrypt unregistered [12545.156748] Key type ._llcrypt unregistered [12560.437539] Key type ._llcrypt registered [12560.445844] Key type .llcrypt registered [12560.543880] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [12561.353881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12561.389671] alg: No test for adler32 (adler32-zlib) [12562.409964] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [12562.610901] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [12564.312959] Key type lgssc registered [12565.425390] Lustre: Echo OBD driver; http://www.lustre.org/ [12591.590518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12591.646846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12592.943675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12593.004091] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12593.150280] Lustre: lustre-MDT0000: new disk, initializing [12593.275662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12593.319755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12597.828570] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [12607.964735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12608.097225] Lustre: 443822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12608.148760] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12608.151979] Lustre: Skipped 1 previous similar message [12608.206638] Lustre: lustre-MDT0001: new disk, initializing [12608.289152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12608.331247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12608.342092] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12612.316245] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [12620.141690] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12624.740565] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12630.318462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12630.573651] Lustre: lustre-OST0000: new disk, initializing [12630.579072] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12630.584235] Lustre: 445791:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12630.698572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12636.092678] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [12640.821084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12640.833094] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12640.956962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12642.944567] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12649.444614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12649.454293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12649.464348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12651.491712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12651.493990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12651.500738] Lustre: Skipped 1 previous similar message [12653.930983] Lustre: server umount lustre-MDT0000 complete [12656.881818] LustreError: 445801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789608713 with bad export cookie 14041760697966209396 [12656.887141] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12656.894121] LustreError: 445801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12657.181464] Lustre: server umount lustre-MDT0001 complete [12670.640170] Lustre: server umount lustre-OST0000 complete [12680.182940] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 21:32:16 (1789608736) [12686.097287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12686.497745] LustreError: 447815:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12686.519893] LustreError: 447815:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12686.566921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12690.721940] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [12691.941248] LustreError: 447816:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12696.054247] LustreError: 447815:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12697.851812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12698.419478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12703.431620] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [12710.479685] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12715.123493] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12733.294201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12733.435084] Lustre: Found index 0 for lustre-OST0000, updating log [12733.566202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12738.551454] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [12744.287781] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12757.103521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12765.665579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12765.670525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12765.686914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12765.691174] Lustre: Skipped 2 previous similar messages [12769.408391] Lustre: server umount lustre-OST0000 complete [12785.634095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12785.642903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12785.653816] Lustre: Skipped 1 previous similar message [12785.658196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12785.665380] Lustre: Skipped 3 previous similar messages [12790.319799] Lustre: server umount lustre-MDT0000 complete [12792.290510] LustreError: 447811:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12792.304457] LustreError: 447811:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12793.060133] LustreError: 450684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789608850 with bad export cookie 14041760697966210789 [12793.060966] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12793.066031] LustreError: 450684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12793.300674] Lustre: server umount lustre-MDT0001 complete [12799.200085] LNet: 1 local NIs in recovery (showing 1): 192.168.206.159@tcp [12800.931468] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 21:34:17 (1789608857) [13415.314063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13415.652522] LustreError: 454247:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13415.735443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13419.656346] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [13421.026570] LustreError: 454248:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13425.120974] LustreError: 454247:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13427.758681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13428.199567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13432.448242] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [13439.110915] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13443.603683] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13461.801869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13462.207918] Lustre: Found index 0 for lustre-OST0000, updating log [13462.519800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13463.538747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [13465.824409] LNet: 1 local NIs in recovery (showing 1): 192.168.206.159@tcp [13468.717708] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [13475.817923] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13478.881322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13478.886245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13478.904272] Lustre: Skipped 1 previous similar message [13478.912800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13478.922832] Lustre: Skipped 2 previous similar messages [13492.192345] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13492.352415] Lustre: server umount lustre-OST0000 complete [13496.166575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13496.174516] Lustre: Skipped 1 previous similar message [13496.174844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13496.180565] Lustre: Skipped 1 previous similar message [13497.185500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13499.234590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13499.238768] Lustre: Skipped 4 previous similar messages [13501.426553] Lustre: server umount lustre-MDT0000 complete [13502.306995] LustreError: 455491:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13502.321120] LustreError: 455491:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13504.261671] LustreError: 454228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789609561 with bad export cookie 14041760697966211874 [13504.268028] LustreError: 454228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13504.268096] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13504.485697] Lustre: server umount lustre-MDT0001 complete [13525.217319] LNet: 1 local NIs in recovery (showing 1): 192.168.206.159@tcp [13531.325975] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [13541.226288] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [13581.232868] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 21:47:17 (1789609637) [13600.111126] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [13602.752571] Key type lgssc unregistered [13603.000653] LNet: 460865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13603.020295] LNetError: 460865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13722.857792] LNet: Removed LNI 192.168.206.159@tcp [13723.686889] Key type .llcrypt unregistered [13723.688463] Key type ._llcrypt unregistered [13740.940546] Key type ._llcrypt registered [13740.942664] Key type .llcrypt registered [13741.057910] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [13742.463702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13742.486697] alg: No test for adler32 (adler32-zlib) [13743.511242] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [13743.762275] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [13745.424227] Key type lgssc registered [13746.411138] Lustre: Echo OBD driver; http://www.lustre.org/ [13749.668459] LNet: Added LNI 192.168.206.159@tcp100 [8/256/0/180] [13750.309789] LNet: Added LNI 192.168.206.159@tcp101 [8/256/0/180] [13750.937582] LNet: Added LNI 192.168.206.159@tcp102 [8/256/0/180] [13751.581582] LNet: Added LNI 192.168.206.159@tcp103 [8/256/0/180] [13752.269667] LNet: Added LNI 192.168.206.159@tcp104 [8/256/0/180] [13752.917136] LNet: Added LNI 192.168.206.159@tcp105 [8/256/0/180] [13753.440474] LNet: Added LNI 192.168.206.159@tcp106 [8/256/0/180] [13754.083066] LNet: Added LNI 192.168.206.159@tcp107 [8/256/0/180] [13754.673121] LNet: Added LNI 192.168.206.159@tcp108 [8/256/0/180] [13755.424851] LNet: Added LNI 192.168.206.159@tcp109 [8/256/0/180] [13756.350564] LNet: Added LNI 192.168.206.159@tcp110 [8/256/0/180] [13756.944679] LNet: Added LNI 192.168.206.159@tcp111 [8/256/0/180] [13757.794604] LNet: Added LNI 192.168.206.159@tcp112 [8/256/0/180] [13758.729497] LNet: Added LNI 192.168.206.159@tcp113 [8/256/0/180] [13759.607975] LNet: Added LNI 192.168.206.159@tcp114 [8/256/0/180] [13760.753797] LNet: Added LNI 192.168.206.159@tcp115 [8/256/0/180] [13762.269511] LNet: Added LNI 192.168.206.159@tcp116 [8/256/0/180] [13763.283277] LNet: Added LNI 192.168.206.159@tcp117 [8/256/0/180] [13764.370607] LNet: Added LNI 192.168.206.159@tcp118 [8/256/0/180] [13765.223213] LNet: Added LNI 192.168.206.159@tcp119 [8/256/0/180] [13766.053977] LNet: Added LNI 192.168.206.159@tcp120 [8/256/0/180] [13767.110694] LNet: Added LNI 192.168.206.159@tcp121 [8/256/0/180] [13768.048108] LNet: Added LNI 192.168.206.159@tcp122 [8/256/0/180] [13768.965589] LNet: Added LNI 192.168.206.159@tcp123 [8/256/0/180] [13769.654756] LNet: Added LNI 192.168.206.159@tcp124 [8/256/0/180] [13770.346273] LNet: Added LNI 192.168.206.159@tcp125 [8/256/0/180] [13771.044711] LNet: Added LNI 192.168.206.159@tcp126 [8/256/0/180] [13771.858915] LNet: Added LNI 192.168.206.159@tcp127 [8/256/0/180] [13772.549928] LNet: Added LNI 192.168.206.159@tcp128 [8/256/0/180] [13773.237703] LNet: Added LNI 192.168.206.159@tcp129 [8/256/0/180] [13773.987559] LNet: Added LNI 192.168.206.159@tcp130 [8/256/0/180] [13774.695925] LNet: Added LNI 192.168.206.159@tcp131 [8/256/0/180] [13775.598721] LNet: Added LNI 192.168.206.159@tcp132 [8/256/0/180] [13776.552800] LNet: Added LNI 192.168.206.159@tcp133 [8/256/0/180] [13777.536951] LNet: Added LNI 192.168.206.159@tcp134 [8/256/0/180] [13778.289727] LNet: Added LNI 192.168.206.159@tcp135 [8/256/0/180] [13785.294520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13785.348032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13786.465473] Lustre: 463894:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.159@tcp: can't add NID 192.168.206.159@tcp130: rc = -75 [13786.569597] LNetError: 461353:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.159 [13786.613918] LNetError: Protocol error connecting to 192.168.206.159@tcp119 at host 192.168.206.159:988: is it running a compatible version of Lustre? [13786.628104] LNetError: Skipped 4 previous similar messages [13786.682786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13786.726668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13786.784162] LNet: 2 local NIs in recovery (showing 2): 192.168.206.159@tcp130, 192.168.206.159@tcp119 [13786.830691] Lustre: lustre-MDT0000: new disk, initializing [13786.899692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13786.924428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13787.681869] LNetError: 463929:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.159 [13787.699133] LNetError: 463929:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [13787.718821] LNetError: Protocol error connecting to 192.168.206.159@tcp110 at host 192.168.206.159:988: is it running a compatible version of Lustre? [13787.741046] LNetError: Skipped 1 previous similar message [13789.857136] LNetError: 463917:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.159 [13789.866178] LNetError: 463917:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [13789.878294] LNetError: Protocol error connecting to 192.168.206.159@tcp130 at host 192.168.206.159:988: is it running a compatible version of Lustre? [13789.891603] LNetError: Skipped 3 previous similar messages [13791.489654] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [13793.954523] LNetError: 463931:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.159 [13793.963233] LNetError: Protocol error connecting to 192.168.206.159@tcp130 at host 192.168.206.159:988: is it running a compatible version of Lustre? [13801.124457] LNetError: 463919:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.159 [13801.133620] LNetError: Protocol error connecting to 192.168.206.159@tcp130 at host 192.168.206.159:988: is it running a compatible version of Lustre? [13802.815435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13802.986206] Lustre: 463910:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13803.046305] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13803.051286] Lustre: Skipped 1 previous similar message [13803.138783] Lustre: lustre-MDT0001: new disk, initializing [13803.196096] Lustre: 464715:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.159@tcp: can't add NID 192.168.206.159@tcp130: rc = -75 [13803.201256] Lustre: 464715:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13803.297681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13803.334112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13803.344281] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13808.013655] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [13810.341741] LNetError: 461350:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.159 [13810.354544] LNetError: 461350:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 12 previous similar messages [13810.363597] LNetError: Protocol error connecting to 192.168.206.159@tcp133 at host 192.168.206.159:988: is it running a compatible version of Lustre? [13810.372068] LNetError: Skipped 12 previous similar messages [13812.266846] Lustre: 465141:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.159@tcp: can't add NID 192.168.206.159@tcp130: rc = -75 [13812.280928] Lustre: 465141:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13817.095903] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13822.830416] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13827.751738] LNetError: 463929:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.159 [13827.768191] LNetError: 463929:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [13827.787463] LNetError: Protocol error connecting to 192.168.206.159@tcp125 at host 192.168.206.159:988: is it running a compatible version of Lustre? [13827.810555] LNetError: Skipped 7 previous similar messages [13829.524721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13829.819466] Lustre: lustre-OST0000: new disk, initializing [13829.824570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13829.835926] Lustre: 465887:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13829.955446] Lustre: 465892:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.159@tcp: can't add NID 192.168.206.159@tcp130: rc = -75 [13829.966510] Lustre: 465892:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13830.041863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13836.120560] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [13838.454127] Lustre: 466344:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.159@tcp: can't add NID 192.168.206.159@tcp130: rc = -75 [13838.459053] Lustre: 466344:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13838.526616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13838.552551] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13838.685357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13846.176202] LNet: 2 local NIs in recovery (showing 2): 192.168.206.159@tcp133, 192.168.206.159@tcp130 [13847.514703] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13859.308803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13859.309348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13859.311696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13859.333672] Lustre: Skipped 1 previous similar message [13860.513173] LNetError: 463931:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.159 [13860.525888] LNetError: 463931:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 16 previous similar messages [13860.536472] LNetError: Protocol error connecting to 192.168.206.159@tcp124 at host 192.168.206.159:988: is it running a compatible version of Lustre? [13860.551378] LNetError: Skipped 16 previous similar messages [13862.282307] Lustre: server umount lustre-OST0000 complete [13869.538856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13869.540317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13869.559132] Lustre: Skipped 1 previous similar message [13869.562251] Lustre: Skipped 1 previous similar message [13871.871202] Lustre: server umount lustre-MDT0000 complete [13875.053241] LustreError: 463903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789609932 with bad export cookie 2581114769671429211 [13875.053718] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13875.059711] LustreError: 463903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13875.327089] Lustre: server umount lustre-MDT0001 complete [13884.797877] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [13887.324214] Key type lgssc unregistered [13887.708569] LNet: 467803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13887.722325] LNetError: 467803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13887.736080] LNet: Removed LNI 192.168.206.159@tcp [13888.800157] LNet: Removed LNI 192.168.206.159@tcp100 [13888.804089] LNetError: 467803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13888.807849] LNetError: 467803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13888.812992] LNet: Removed LNI 192.168.206.159@tcp101 [13889.888092] LNet: Removed LNI 192.168.206.159@tcp102 [13889.898909] LNetError: 467803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13889.904366] LNetError: 467803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13889.909458] LNet: Removed LNI 192.168.206.159@tcp103 [13890.979257] LNet: Removed LNI 192.168.206.159@tcp104 [13892.003983] LNet: Removed LNI 192.168.206.159@tcp105 [13892.013917] LNetError: 467803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13892.025799] LNetError: 467803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13893.088302] LNet: Removed LNI 192.168.206.159@tcp106 [13894.112816] LNet: Removed LNI 192.168.206.159@tcp107 [13895.136147] LNet: Removed LNI 192.168.206.159@tcp108 [13896.160601] LNet: Removed LNI 192.168.206.159@tcp109 [13896.166043] LNetError: 467803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13896.172543] LNetError: 467803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [13896.189792] LNet: Removed LNI 192.168.206.159@tcp110 [13897.248162] LNet: Removed LNI 192.168.206.159@tcp111 [13898.272735] LNet: Removed LNI 192.168.206.159@tcp112 [13899.296105] LNet: Removed LNI 192.168.206.159@tcp113 [13900.320121] LNet: Removed LNI 192.168.206.159@tcp114 [13901.348724] LNet: Removed LNI 192.168.206.159@tcp115 [13902.368656] LNet: Removed LNI 192.168.206.159@tcp116 [13903.392792] LNet: Removed LNI 192.168.206.159@tcp117 [13904.416342] LNet: Removed LNI 192.168.206.159@tcp118 [13904.418165] LNetError: 467803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13904.427785] LNetError: 467803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13904.432624] LNet: Removed LNI 192.168.206.159@tcp119 [13905.440168] LNet: Removed LNI 192.168.206.159@tcp120 [13906.464147] LNet: Removed LNI 192.168.206.159@tcp121 [13907.488245] LNet: Removed LNI 192.168.206.159@tcp122 [13908.512571] LNet: Removed LNI 192.168.206.159@tcp123 [13909.536405] LNet: Removed LNI 192.168.206.159@tcp124 [13910.562202] LNet: Removed LNI 192.168.206.159@tcp125 [13911.584353] LNet: Removed LNI 192.168.206.159@tcp126 [13912.608162] LNet: Removed LNI 192.168.206.159@tcp127 [13913.632230] LNet: Removed LNI 192.168.206.159@tcp128 [13914.657144] LNet: Removed LNI 192.168.206.159@tcp129 [13914.661974] LNet: Removed LNI 192.168.206.159@tcp130 [13915.680194] LNet: Removed LNI 192.168.206.159@tcp131 [13916.704484] LNet: Removed LNI 192.168.206.159@tcp132 [13916.707127] LNet: Removed LNI 192.168.206.159@tcp133 [13917.728167] LNet: Removed LNI 192.168.206.159@tcp134 [13918.760802] LNet: Removed LNI 192.168.206.159@tcp135 [13919.546374] Key type .llcrypt unregistered [13919.549155] Key type ._llcrypt unregistered [13940.214583] Key type ._llcrypt registered [13940.216945] Key type .llcrypt registered [13940.292799] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [13953.290287] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [13954.079947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13954.194380] alg: No test for adler32 (adler32-zlib) [13955.163344] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [13955.353033] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [13956.984210] Key type lgssc registered [13957.949686] Lustre: Echo OBD driver; http://www.lustre.org/ [13989.978200] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 21:54:06 (1789610046) [14008.288716] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [14010.474130] Key type lgssc unregistered [14010.745848] LNet: 471887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14010.761232] LNetError: 471887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14010.780377] LNet: Removed LNI 192.168.206.159@tcp [14011.510867] Key type .llcrypt unregistered [14011.516181] Key type ._llcrypt unregistered [14027.694662] Key type ._llcrypt registered [14027.695892] Key type .llcrypt registered [14027.797712] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [14028.384391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14028.442531] alg: No test for adler32 (adler32-zlib) [14029.370039] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [14029.518913] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [14031.184551] Key type lgssc registered [14032.111236] Lustre: Echo OBD driver; http://www.lustre.org/ [14041.089764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14041.153425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14042.443995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14042.482410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14042.634277] Lustre: lustre-MDT0000: new disk, initializing [14042.755594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14042.805927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14048.125675] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [14058.492060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14058.556603] Lustre: 473095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14058.608137] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14058.612219] Lustre: Skipped 1 previous similar message [14058.682321] Lustre: lustre-MDT0001: new disk, initializing [14058.762876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14058.820145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14058.831467] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14063.143601] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [14071.940593] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14076.831620] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14082.421240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14082.644821] Lustre: lustre-OST0000: new disk, initializing [14082.647470] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14082.652730] Lustre: 475067:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14082.718512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14087.689596] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [14089.257846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14089.268419] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14089.339171] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14096.187961] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14099.395879] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing lnet_if_list [14103.915317] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14103.931207] LNet: Added LNI 192.168.206.159@tcp10 [8/256/0/180] [14109.491360] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14109.536440] LNet: Added LNI 192.168.206.159@tcp11 [8/256/0/180] [14113.646861] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14113.670816] LNet: Added LNI 192.168.206.159@tcp12 [8/256/0/180] [14117.689573] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14117.734170] LNet: Added LNI 192.168.206.159@tcp13 [8/256/0/180] [14121.792824] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14121.816952] LNet: Added LNI 192.168.206.159@tcp14 [8/256/0/180] [14125.718522] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14125.741423] LNet: Added LNI 192.168.206.159@tcp15 [8/256/0/180] [14130.197493] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14130.225484] LNet: Added LNI 192.168.206.159@tcp16 [8/256/0/180] [14134.926811] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14134.955704] LNet: Added LNI 192.168.206.159@tcp17 [8/256/0/180] [14139.922873] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14139.953997] LNet: Added LNI 192.168.206.159@tcp18 [8/256/0/180] [14144.872373] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14144.916580] LNet: Added LNI 192.168.206.159@tcp19 [8/256/0/180] [14150.542719] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14150.577601] LNet: Added LNI 192.168.206.159@tcp20 [8/256/0/180] [14156.145833] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14156.166668] LNet: Added LNI 192.168.206.159@tcp21 [8/256/0/180] [14162.251494] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14162.277665] LNet: Added LNI 192.168.206.159@tcp22 [8/256/0/180] [14167.583563] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14167.614992] LNet: Added LNI 192.168.206.159@tcp23 [8/256/0/180] [14172.409126] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14172.447718] LNet: Added LNI 192.168.206.159@tcp24 [8/256/0/180] [14177.519705] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14177.554225] LNet: Added LNI 192.168.206.159@tcp25 [8/256/0/180] [14183.228586] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14183.283638] LNet: Added LNI 192.168.206.159@tcp26 [8/256/0/180] [14188.818350] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14188.846680] LNet: Added LNI 192.168.206.159@tcp27 [8/256/0/180] [14194.938220] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14194.961030] LNet: Added LNI 192.168.206.159@tcp28 [8/256/0/180] [14200.968864] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14200.996109] LNet: Added LNI 192.168.206.159@tcp29 [8/256/0/180] [14206.829634] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14206.868604] LNet: Added LNI 192.168.206.159@tcp30 [8/256/0/180] [14212.846892] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14212.877473] LNet: Added LNI 192.168.206.159@tcp31 [8/256/0/180] [14219.340990] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14219.373080] LNet: Added LNI 192.168.206.159@tcp32 [8/256/0/180] [14225.702844] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14225.739821] LNet: Added LNI 192.168.206.159@tcp33 [8/256/0/180] [14233.032153] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14233.065703] LNet: Added LNI 192.168.206.159@tcp34 [8/256/0/180] [14240.000523] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14240.032806] LNet: Added LNI 192.168.206.159@tcp35 [8/256/0/180] [14246.576681] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14246.622231] LNet: Added LNI 192.168.206.159@tcp36 [8/256/0/180] [14252.985789] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14253.011283] LNet: Added LNI 192.168.206.159@tcp37 [8/256/0/180] [14259.503658] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14259.533969] LNet: Added LNI 192.168.206.159@tcp38 [8/256/0/180] [14265.996060] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14266.036147] LNet: Added LNI 192.168.206.159@tcp39 [8/256/0/180] [14273.656468] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14273.679489] LNet: Added LNI 192.168.206.159@tcp40 [8/256/0/180] [14281.446354] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14281.493261] LNet: Added LNI 192.168.206.159@tcp41 [8/256/0/180] [14288.352742] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14288.376899] LNet: Added LNI 192.168.206.159@tcp42 [8/256/0/180] [14294.845085] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14294.867337] LNet: Added LNI 192.168.206.159@tcp43 [8/256/0/180] [14301.502670] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14301.531613] LNet: Added LNI 192.168.206.159@tcp44 [8/256/0/180] [14307.659554] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14307.701611] LNet: Added LNI 192.168.206.159@tcp45 [8/256/0/180] [14315.378290] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14315.401578] LNet: Added LNI 192.168.206.159@tcp46 [8/256/0/180] [14322.774026] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14322.810793] LNet: Added LNI 192.168.206.159@tcp47 [8/256/0/180] [14330.391599] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14330.440612] LNet: Added LNI 192.168.206.159@tcp48 [8/256/0/180] [14338.422561] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14338.464807] LNet: Added LNI 192.168.206.159@tcp49 [8/256/0/180] [14348.388332] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14348.437637] LNet: Added LNI 192.168.206.159@tcp50 [8/256/0/180] [14370.795359] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14370.798319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14370.812565] Lustre: Skipped 2 previous similar messages [14370.815537] Lustre: Skipped 1 previous similar message [14373.879298] Lustre: server umount lustre-MDT0000 complete [14378.577049] LustreError: 473891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789610435 with bad export cookie 10730847910154256264 [14378.579964] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14378.591349] LustreError: 473891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14379.002442] Lustre: server umount lustre-MDT0001 complete [14397.344114] Lustre: 472366:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789610438/real 1789610438] req@000000005128841a x1876542185700352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789610454 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14397.375193] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14398.003174] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [14399.456119] Lustre: 472368:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789610440/real 1789610440] req@0000000086279882 x1876542185700480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789610456 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14402.528155] Lustre: 472369:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789610443/real 1789610443] req@00000000c3f23905 x1876542185700608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789610459 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14406.941035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14409.696139] Lustre: 472367:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789610450/real 1789610450] req@00000000a185e6aa x1876542185700992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789610466 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14409.731260] Lustre: 472367:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14423.012440] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x94eba151df7cc944 [14423.038076] Lustre: MGC192.168.206.159@tcp: Connection restored to 0@lo (at 0@lo) [14423.501755] LustreError: 475060:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14423.536037] LustreError: 475060:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [14423.620057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14426.597067] LustreError: 475061:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14427.619582] LustreError: 475060:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14427.624866] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14429.077856] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [14432.747790] LustreError: 475061:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14437.867496] LustreError: 475060:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14437.936779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14438.347213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14442.479576] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14443.265874] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [14451.352496] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14456.623965] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14478.312291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14478.343844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14478.359074] Lustre: Skipped 2 previous similar messages [14482.936608] Lustre: server umount lustre-OST0000 complete [14493.668434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14493.685944] Lustre: Skipped 1 previous similar message [14493.699031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14493.706587] Lustre: Skipped 1 previous similar message [14496.090979] Lustre: server umount lustre-MDT0000 complete [14498.788724] LustreError: 499321:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14498.806688] LustreError: 499321:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14499.912392] LustreError: 473089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789610556 with bad export cookie 10730847910154258756 [14499.919541] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14499.935881] LustreError: 473089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14500.177768] Lustre: server umount lustre-MDT0001 complete [14511.798970] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [14515.124548] Key type lgssc unregistered [14515.574976] LNet: 502117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14515.584521] LNetError: 502117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14516.640310] LNet: Removed LNI 192.168.206.159@tcp [14516.642513] LNetError: 502117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14516.649829] LNet: Removed LNI 192.168.206.159@tcp10 [14516.655102] LNet: Removed LNI 192.168.206.159@tcp11 [14516.658892] LNet: Removed LNI 192.168.206.159@tcp12 [14516.661953] LNet: Removed LNI 192.168.206.159@tcp13 [14516.666477] LNet: Removed LNI 192.168.206.159@tcp14 [14516.677978] LNet: Removed LNI 192.168.206.159@tcp15 [14516.681296] LNet: Removed LNI 192.168.206.159@tcp16 [14516.684701] LNet: Removed LNI 192.168.206.159@tcp17 [14516.687573] LNet: Removed LNI 192.168.206.159@tcp18 [14516.690858] LNet: Removed LNI 192.168.206.159@tcp19 [14516.693789] LNet: Removed LNI 192.168.206.159@tcp20 [14516.697070] LNet: Removed LNI 192.168.206.159@tcp21 [14516.700309] LNet: Removed LNI 192.168.206.159@tcp22 [14516.705281] LNet: Removed LNI 192.168.206.159@tcp23 [14516.713218] LNet: Removed LNI 192.168.206.159@tcp24 [14516.718206] LNet: Removed LNI 192.168.206.159@tcp25 [14516.721100] LNet: Removed LNI 192.168.206.159@tcp26 [14516.727249] LNet: Removed LNI 192.168.206.159@tcp27 [14516.732922] LNet: Removed LNI 192.168.206.159@tcp28 [14516.736411] LNet: Removed LNI 192.168.206.159@tcp29 [14516.739839] LNet: Removed LNI 192.168.206.159@tcp30 [14516.743082] LNet: Removed LNI 192.168.206.159@tcp31 [14516.746582] LNet: Removed LNI 192.168.206.159@tcp32 [14516.748736] LNet: Removed LNI 192.168.206.159@tcp33 [14516.751160] LNet: Removed LNI 192.168.206.159@tcp34 [14516.754786] LNet: Removed LNI 192.168.206.159@tcp35 [14516.758700] LNet: Removed LNI 192.168.206.159@tcp36 [14516.765364] LNet: Removed LNI 192.168.206.159@tcp37 [14516.767414] LNet: Removed LNI 192.168.206.159@tcp38 [14516.770120] LNet: Removed LNI 192.168.206.159@tcp39 [14516.773051] LNet: Removed LNI 192.168.206.159@tcp40 [14516.776387] LNet: Removed LNI 192.168.206.159@tcp41 [14516.779525] LNet: Removed LNI 192.168.206.159@tcp42 [14516.782866] LNet: Removed LNI 192.168.206.159@tcp43 [14516.786172] LNet: Removed LNI 192.168.206.159@tcp44 [14516.788174] LNet: Removed LNI 192.168.206.159@tcp45 [14516.790071] LNet: Removed LNI 192.168.206.159@tcp46 [14516.792334] LNet: Removed LNI 192.168.206.159@tcp47 [14516.794897] LNet: Removed LNI 192.168.206.159@tcp48 [14516.797826] LNet: Removed LNI 192.168.206.159@tcp49 [14516.806679] LNet: Removed LNI 192.168.206.159@tcp50 [14517.484649] Key type .llcrypt unregistered [14517.488426] Key type ._llcrypt unregistered [14540.770611] Key type ._llcrypt registered [14540.772685] Key type .llcrypt registered [14540.906838] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [14556.657970] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [14557.691624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14557.712085] alg: No test for adler32 (adler32-zlib) [14558.880128] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [14559.060788] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [14560.688193] Key type lgssc registered [14561.810895] Lustre: Echo OBD driver; http://www.lustre.org/ [14600.325589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14600.363918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14601.545273] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14601.571857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14601.648946] Lustre: lustre-MDT0000: new disk, initializing [14601.718268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14601.736386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14605.225430] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [14616.494357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14616.639639] Lustre: 505312:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14616.690718] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14616.694898] Lustre: Skipped 1 previous similar message [14616.773523] Lustre: lustre-MDT0001: new disk, initializing [14616.869359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14616.908540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14616.937098] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14621.379838] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [14630.548833] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14635.691753] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14642.675254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14642.876771] Lustre: *** cfs_fail_loc=90f, val=0*** [14642.958302] Lustre: lustre-OST0000: new disk, initializing [14642.961821] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14642.968425] Lustre: 507329:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14643.047647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14649.027810] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [14652.501834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14652.516740] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14652.613865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14658.508281] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14668.264041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14668.270293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14668.279260] LustreError: Skipped 1 previous similar message [14668.280695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14668.316563] Lustre: Skipped 1 previous similar message [14673.380820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14673.385213] Lustre: Skipped 2 previous similar messages [14678.502908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14678.514355] Lustre: Skipped 1 previous similar message [14680.032384] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14680.208624] Lustre: server umount lustre-OST0000 complete [14688.739025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14688.740522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14688.747201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14688.780372] Lustre: Skipped 1 previous similar message [14690.649674] Lustre: server umount lustre-MDT0000 complete [14694.508983] LustreError: 505348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789610751 with bad export cookie 717695207073258640 [14694.509690] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14694.529047] LustreError: 505348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14694.789706] Lustre: server umount lustre-MDT0001 complete [14705.100599] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [14707.520758] Key type lgssc unregistered [14707.737890] LNet: 509235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14707.742403] LNetError: 509235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14707.752378] LNet: Removed LNI 192.168.206.159@tcp [14708.543962] Key type .llcrypt unregistered [14708.547107] Key type ._llcrypt unregistered [14727.723490] Key type ._llcrypt registered [14727.729893] Key type .llcrypt registered [14727.895684] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [14729.253155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14729.312248] alg: No test for adler32 (adler32-zlib) [14730.423347] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [14730.780787] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [14732.528276] Key type lgssc registered [14733.879855] Lustre: Echo OBD driver; http://www.lustre.org/ [14741.958933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14741.980243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14743.358929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14746.979128] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [14753.764419] LustreError: 510505:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14753.803585] LustreError: 510505:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [14755.647509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14756.132862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14761.188478] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [14769.778387] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14775.893469] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14782.767595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14783.199635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14789.960375] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [14797.601659] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14813.091585] loop0: detected capacity change from 0 to 204800000 [14813.392906] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14813.419287] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14813.430911] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14820.893817] loop0: detected capacity change from 0 to 204800000 [14821.168051] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14821.177137] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14821.209917] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14827.645832] loop0: detected capacity change from 0 to 204800000 [14834.486652] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14834.800291] Lustre: fs15246-OST0000: new disk, initializing [14834.811911] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14834.823152] Lustre: 514098:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14834.944435] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14841.755313] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [14852.517752] loop1: detected capacity change from 0 to 204800000 [14859.114519] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14859.241464] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14859.312930] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14859.413969] Lustre: fs15246-MDT0000: new disk, initializing [14859.694028] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14859.720830] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14859.938290] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14860.963238] Lustre: 515221:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000492e91a6 x1876542920640256/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789610933 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14865.087771] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14865.101158] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14865.165141] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14865.464773] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [14890.473736] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14890.504078] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14895.480848] Lustre: server umount fs15246-MDT0000 complete [14895.593846] LustreError: 510500:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14900.713518] LustreError: 510500:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14905.834407] LustreError: 516364:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14906.958863] Lustre: server umount fs15246-OST0000 complete [14921.207904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14921.209917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14921.224927] Lustre: Skipped 1 previous similar message [14921.232744] Lustre: Skipped 1 previous similar message [14926.308379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14926.322458] Lustre: Skipped 1 previous similar message [14926.601426] Lustre: server umount lustre-OST0000 complete [14931.425636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14931.435538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14931.455196] Lustre: Skipped 1 previous similar message [14931.467543] Lustre: Skipped 1 previous similar message [14936.551788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14936.560305] Lustre: Skipped 1 previous similar message [14936.924406] Lustre: server umount lustre-MDT0000 complete [14941.252923] LustreError: 510485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789610998 with bad export cookie 13448131917891485447 [14941.254923] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14941.266279] LustreError: 510485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14941.764192] Lustre: server umount lustre-MDT0001 complete [14954.976846] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [14958.119862] Key type lgssc unregistered [14958.439802] LNet: 518319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14958.449763] LNetError: 518319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14958.474299] LNet: Removed LNI 192.168.206.159@tcp [14959.801553] Key type .llcrypt unregistered [14959.805180] Key type ._llcrypt unregistered [14981.671612] Key type ._llcrypt registered [14981.674342] Key type .llcrypt registered [14981.812388] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [14983.087774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14983.217980] alg: No test for adler32 (adler32-zlib) [14984.367773] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [14984.738982] LNet: Added LNI 192.168.206.159@tcp [8/256/0/180] [14986.504195] Key type lgssc registered [14987.992869] Lustre: Echo OBD driver; http://www.lustre.org/ [15035.809275] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [15047.403269] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [15091.655943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15091.695467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15093.059392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15093.104189] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15093.203151] Lustre: lustre-MDT0000: new disk, initializing [15093.280637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15093.304413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15098.212868] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [15109.817386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15110.000846] Lustre: 522790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15110.071896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15110.079138] Lustre: Skipped 1 previous similar message [15110.223553] Lustre: lustre-MDT0001: new disk, initializing [15110.378905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15110.452610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15110.472100] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15115.610482] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [15125.876872] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15132.430522] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15141.041445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15141.556700] Lustre: lustre-OST0000: new disk, initializing [15141.569062] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15141.577211] Lustre: 524759:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15141.688906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15147.122768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15147.139280] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15147.323259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15149.695389] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_default_debug -1 all [15159.603872] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15164.371918] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15164.600762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15168.874515] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15169.049454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15173.091670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15173.096444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15173.102128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15173.131576] Lustre: Skipped 1 previous similar message [15173.141423] Lustre: Skipped 1 previous similar message [15178.212422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15178.238135] Lustre: Skipped 1 previous similar message [15183.331597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15183.347299] Lustre: Skipped 1 previous similar message [15185.377581] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15185.566345] Lustre: server umount lustre-OST0000 complete [15192.545827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15192.566128] LustreError: Skipped 1 previous similar message [15192.573356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15192.593220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15196.929840] Lustre: server umount lustre-MDT0000 complete [15201.506435] LustreError: 522780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789611258 with bad export cookie 6498698571438964423 [15201.519101] LustreError: MGC192.168.206.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15201.521881] LustreError: 522780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15201.968078] Lustre: server umount lustre-MDT0001 complete [15234.562931] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing set_hostid [15246.947125] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing load_modules_local [15285.840550] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14991 sec ========= 22:15:41 (1789611341) [15288.483255] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:15:44 (1789611344) === [15293.748782] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:15:49 (1789611349) === [15336.956815] Lustre: DEBUG MARKER: oleg659-server.virtnet: executing unload_modules_local [15340.506261] Key type lgssc unregistered [15340.830974] LNet: 531406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15340.842374] LNetError: 531406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15340.863411] LNet: Removed LNI 192.168.206.159@tcp [15341.994915] Key type .llcrypt unregistered [15341.998367] Key type ._llcrypt unregistered