[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 470063260 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002385] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009123] LSM: Security Framework initializing [ 0.010051] Yama: becoming mindful. [ 0.011038] SELinux: Initializing. [ 0.012073] *** VALIDATE selinux *** [ 0.021198] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026212] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027145] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029112] *** VALIDATE tmpfs *** [ 0.031236] *** VALIDATE proc *** [ 0.032234] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.034252] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037030] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.041097] debug: unmapping init [mem 0xffffffffbb659000-0xffffffffbb660fff] [ 0.044162] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045718] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046020] ... version: 2 [ 0.047010] ... bit width: 48 [ 0.048010] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050013] ... max period: 00007fffffffffff [ 0.051014] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053299] rcu: Hierarchical SRCU implementation. [ 0.055448] smp: Bringing up secondary CPUs ... [ 0.056650] x86: Booting SMP configuration: [ 0.057025] .... node #0, CPUs: #1 #2 #3 [ 0.061413] smp: Brought up 1 node, 4 CPUs [ 0.063012] smpboot: Max logical packages: 1 [ 0.064016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142691] node 0 deferred pages initialised in 75ms [ 0.146178] devtmpfs: initialized [ 0.147240] x86/mm: Memory block size: 128MB [ 0.149806] gcov: version magic: 0x41383552 [ 0.151147] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152102] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153254] pinctrl core: initialized pinctrl subsystem [ 0.154219] [ 0.154829] ************************************************************* [ 0.155013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156013] ** ** [ 0.157021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158014] ** ** [ 0.159013] ** This means that this kernel is built to expose internal ** [ 0.160013] ** IOMMU data structures, which may compromise security on ** [ 0.161016] ** your system. ** [ 0.162012] ** ** [ 0.163017] ** If you see this message and you are not debugging the ** [ 0.164015] ** kernel, report this immediately to your vendor! ** [ 0.165014] ** ** [ 0.166012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167013] ************************************************************* [ 0.168829] NET: Registered protocol family 16 [ 0.169450] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.170068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.171063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.173038] cpuidle: using governor menu [ 0.174708] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.178448] PCI: Using configuration type 1 for base access [ 0.181134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.190138] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.193020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.197147] cryptd: max_cpu_qlen set to 1000 [ 0.199289] ACPI: Added _OSI(Module Device) [ 0.201014] ACPI: Added _OSI(Processor Device) [ 0.203012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.205013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.212052] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.218350] ACPI: Interpreter enabled [ 0.220081] ACPI: PM: (supports S0 S3 S4 S5) [ 0.222011] ACPI: Using IOAPIC for interrupt routing [ 0.223096] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.227340] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.238463] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.241042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.243019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.247150] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.253010] acpiphp: Slot [2] registered [ 0.254130] acpiphp: Slot [5] registered [ 0.256125] acpiphp: Slot [6] registered [ 0.258247] acpiphp: Slot [7] registered [ 0.260138] acpiphp: Slot [8] registered [ 0.262109] acpiphp: Slot [9] registered [ 0.263064] acpiphp: Slot [10] registered [ 0.264100] acpiphp: Slot [3] registered [ 0.266097] acpiphp: Slot [4] registered [ 0.267094] acpiphp: Slot [11] registered [ 0.269125] acpiphp: Slot [12] registered [ 0.270100] acpiphp: Slot [13] registered [ 0.272117] acpiphp: Slot [14] registered [ 0.274095] acpiphp: Slot [15] registered [ 0.275108] acpiphp: Slot [16] registered [ 0.277172] acpiphp: Slot [17] registered [ 0.279106] acpiphp: Slot [18] registered [ 0.281113] acpiphp: Slot [19] registered [ 0.282082] acpiphp: Slot [20] registered [ 0.284112] acpiphp: Slot [21] registered [ 0.286079] acpiphp: Slot [22] registered [ 0.287112] acpiphp: Slot [23] registered [ 0.289117] acpiphp: Slot [24] registered [ 0.291360] acpiphp: Slot [25] registered [ 0.292095] acpiphp: Slot [26] registered [ 0.294138] acpiphp: Slot [27] registered [ 0.296108] acpiphp: Slot [28] registered [ 0.297108] acpiphp: Slot [29] registered [ 0.299091] acpiphp: Slot [30] registered [ 0.300154] acpiphp: Slot [31] registered [ 0.302058] PCI host bridge to bus 0000:00 [ 0.303016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.306021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.308020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.311022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.313019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.315031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.317188] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.321066] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.324594] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.337013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.342412] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.346026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.350018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.353018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.356484] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.358824] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.361054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.364683] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.369013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.381010] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.386013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.392742] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.403015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.410014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.435017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.445405] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.456013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.464014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.493014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.510877] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.523013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.533012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.556024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.565976] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.579015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.587013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.605017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.616917] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.626024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.633022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.655029] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.667201] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.673015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.681014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.708018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.721231] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.723389] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.726378] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.728404] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.731234] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.737109] iommu: Default domain type: Passthrough [ 0.739475] SCSI subsystem initialized [ 0.741161] ACPI: bus type USB registered [ 0.742108] usbcore: registered new interface driver usbfs [ 0.744075] usbcore: registered new interface driver hub [ 0.746091] usbcore: registered new device driver usb [ 0.748175] pps_core: LinuxPPS API ver. 1 registered [ 0.750011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.752055] PTP clock support registered [ 0.754142] EDAC MC: Ver: 3.0.0 [ 0.755557] PCI: Using ACPI for IRQ routing [ 0.757958] NetLabel: Initializing [ 0.760013] NetLabel: domain hash size = 128 [ 0.761009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.763078] NetLabel: unlabeled traffic allowed by default [ 0.766052] vgaarb: loaded [ 0.768042] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.770013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.776292] clocksource: Switched to clocksource kvm-clock [ 0.879410] VFS: Disk quotas dquot_6.6.0 [ 0.881157] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.884167] *** VALIDATE ramfs *** [ 0.885481] *** VALIDATE hugetlbfs *** [ 0.887117] pnp: PnP ACPI init [ 0.889460] pnp: PnP ACPI: found 6 devices [ 0.905844] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.909890] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.912098] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.914615] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.916931] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.919543] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.923355] NET: Registered protocol family 2 [ 0.925962] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.930435] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.933467] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.938305] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.942086] TCP: Hash tables configured (established 65536 bind 65536) [ 0.944992] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.948217] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.950830] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.954250] NET: Registered protocol family 1 [ 0.957361] RPC: Registered named UNIX socket transport module. [ 0.960333] RPC: Registered udp transport module. [ 0.962267] RPC: Registered tcp transport module. [ 0.964281] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.966979] NET: Registered protocol family 44 [ 0.968640] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.970797] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.972853] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.975108] PCI: CLS 0 bytes, default 64 [ 0.976373] Unpacking initramfs... [ 2.345380] debug: unmapping init [mem 0xffff9bf2bcc54000-0xffff9bf2bffbffff] [ 2.351346] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.353883] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.357264] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.850809] Initialise system trusted keyrings [ 2.852593] Key type blacklist registered [ 2.854612] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.863692] zbud: loaded [ 2.867099] *** VALIDATE nfs *** [ 2.867995] *** VALIDATE nfs4 *** [ 2.870035] pstore: using deflate compression [ 2.874182] Platform Keyring initialized [ 2.974394] NET: Registered protocol family 38 [ 2.976296] Key type asymmetric registered [ 2.977602] Asymmetric key parser 'x509' registered [ 2.980241] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.983445] io scheduler mq-deadline registered [ 2.984972] io scheduler kyber registered [ 2.987263] io scheduler bfq registered [ 2.988885] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.991425] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.994652] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.997555] ACPI: Power Button [PWRF] [ 3.003357] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.013041] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.034973] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.042842] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.067493] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.097445] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.126668] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.132682] Non-volatile memory driver v1.3 [ 3.135348] Linux agpgart interface v0.103 [ 3.171102] virtio_blk virtio1: [vda] 136000 512-byte logical blocks (69.6 MB/66.4 MiB) [ 3.173920] vda: detected capacity change from 0 to 69632000 [ 3.191643] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.194773] vdb: detected capacity change from 0 to 1073741824 [ 3.221701] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.225561] vdc: detected capacity change from 0 to 2621440000 [ 3.241335] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.244581] vdd: detected capacity change from 0 to 2621440000 [ 3.258865] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.262109] vde: detected capacity change from 0 to 4294967296 [ 3.280173] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.283228] vdf: detected capacity change from 0 to 4294967296 [ 3.291347] libphy: Fixed MDIO Bus: probed [ 3.296273] usbcore: registered new interface driver usbserial_generic [ 3.298916] usbserial: USB Serial support registered for generic [ 3.301846] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.307365] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.309606] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.312538] mousedev: PS/2 mouse device common for all mice [ 3.315986] rtc_cmos 00:05: RTC can wake from S4 [ 3.319137] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.324773] rtc_cmos 00:05: registered as rtc0 [ 3.326721] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.327511] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.329449] intel_pstate: CPU model not supported [ 3.336665] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.336712] hid: raw HID events driver (C) Jiri Kosina [ 3.343362] usbcore: registered new interface driver usbhid [ 3.345429] usbhid: USB HID core driver [ 3.346960] drop_monitor: Initializing network drop monitor service [ 3.350291] Initializing XFRM netlink socket [ 3.352196] NET: Registered protocol family 10 [ 3.355392] Segment Routing with IPv6 [ 3.356958] NET: Registered protocol family 17 [ 3.359194] mpls_gso: MPLS GSO support [ 3.366122] RAS: Correctable Errors collector initialized. [ 3.369076] AVX version of gcm_enc/dec engaged. [ 3.373227] AES CTR mode by8 optimization enabled [ 3.445699] sched_clock: Marking stable (3445669189, 0)->(4383026885, -937357696) [ 3.449121] registered taskstats version 1 [ 3.451135] Loading compiled-in X.509 certificates [ 3.453333] zswap: loaded using pool lzo/zbud [ 3.476715] Key type big_key registered [ 3.491354] Key type encrypted registered [ 3.493028] ima: No TPM chip found, activating TPM-bypass! [ 3.495142] ima: Allocated hash algorithm: sha1 [ 3.496995] ima: No architecture policies found [ 3.498861] evm: Initialising EVM extended attributes: [ 3.500849] evm: security.selinux [ 3.502116] evm: security.ima [ 3.503180] evm: security.capability [ 3.504612] evm: HMAC attrs: 0x1 [ 3.507143] rtc_cmos 00:05: setting system clock to 2026-06-03 01:30:33 UTC (1780450233) [ 3.512232] debug: unmapping init [mem 0xffffffffbc603000-0xffffffffbc7fffff] [ 3.514864] debug: unmapping init [mem 0xffffffffbb382000-0xffffffffbb658fff] [ 3.525459] Write protecting the kernel read-only data: 28672k [ 3.528576] debug: unmapping init [mem 0xffffffffb9a03000-0xffffffffb9bfffff] [ 3.531490] debug: unmapping init [mem 0xffffffffba314000-0xffffffffba3fffff] [ 3.562569] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.572357] systemd[1]: Detected virtualization kvm. [ 3.574730] systemd[1]: Detected architecture x86-64. [ 3.577243] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.607735] systemd[1]: No hostname configured. [ 3.610116] systemd[1]: Set hostname to . [ 3.612862] random: systemd: uninitialized urandom read (16 bytes read) [ 3.615441] systemd[1]: Initializing machine ID from random generator. [ 3.673390] random: ln: uninitialized urandom read (6 bytes read) [ 3.753551] random: systemd: uninitialized urandom read (16 bytes read) [ 3.756499] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.762783] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.767372] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.453318] device-mapper: uevent: version 1.0.3 [ 4.455862] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.293143] random: fast init done [ 5.329428] virtio_net virtio0 ens2: renamed from eth0 [ 5.397539] scsi host0: ata_piix [ 5.409443] scsi host1: ata_piix [ 5.411321] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.413547] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.265555] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.011560] random: crng init done [ 10.013454] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.594754] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.822991] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.084531] SELinux: Disabled at runtime. [ 12.152760] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.162779] systemd[1]: Detected virtualization kvm. [ 12.165151] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.688789] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.692457] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.698246] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.703369] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.707494] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.716902] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.721455] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ 12.843267] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.194177] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.585979] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.673724] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.778415] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.790061] EDAC sbridge: Ver: 1.1.2 [ 15.474081] Key type dns_resolver registered [ 15.812518] NFS: Registering the id_resolver key type [ 15.814661] Key type id_resolver registered [ 15.816503] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ 22.526359] hrtimer: interrupt took 3350662 ns Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg354-server login: [ 66.239885] libcfs: loading out-of-tree module taints kernel. [ 66.314347] Key type ._llcrypt registered [ 66.320580] Key type .llcrypt registered [ 66.419387] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 86.469222] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 88.226909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 88.271504] alg: No test for adler32 (adler32-zlib) [ 90.150977] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 91.285234] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 93.199271] Key type lgssc registered [ 95.537754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 116.037254] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 161.382307] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 175.173236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 175.232433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 176.626108] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 176.666869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 176.807261] Lustre: lustre-MDT0000: new disk, initializing [ 176.955306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 176.973959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 182.130959] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.876313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.939555] Lustre: 6501:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 196.969642] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 196.973892] Lustre: Skipped 1 previous similar message [ 197.068934] Lustre: lustre-MDT0001: new disk, initializing [ 197.171449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 197.224733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 197.255231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 201.656140] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.762942] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 217.282137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 217.555919] Lustre: lustre-OST0000: new disk, initializing [ 217.564730] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 217.580825] Lustre: 8407:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 217.669949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 224.167982] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.905258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 227.922218] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 228.014935] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 240.211676] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.469675] Lustre: lustre-OST0001: new disk, initializing [ 240.476154] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 240.487707] Lustre: 9462:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 240.548610] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 246.132865] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 246.810323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 246.815524] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 246.872355] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 256.449462] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 265.522792] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 271.722874] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing check_logdir /tmp/testlogs/ [ 276.426798] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing yml_node [ 280.003504] Lustre: DEBUG MARKER: Client: 2.17.53.57 [ 282.485721] Lustre: DEBUG MARKER: MDS: 2.17.53.57 [ 285.026932] Lustre: DEBUG MARKER: OSS: 2.17.53.57 [ 286.628485] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 2 21:35:15 EDT 2026 [ 303.910578] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 305.248502] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 323.552459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.553781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 323.564848] Lustre: Skipped 3 previous similar messages [ 323.576896] Lustre: Skipped 3 previous similar messages [ 326.794373] Lustre: server umount lustre-MDT0000 complete [ 330.515730] LustreError: 10259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780450560 with bad export cookie 12554079948543893911 [ 330.518632] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 330.537271] LustreError: 10259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 333.793162] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.812606] LustreError: 6510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.815420] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 333.832215] LustreError: 6510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 333.854174] Lustre: Skipped 1 previous similar message [ 335.842966] LustreError: 8866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 335.843473] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 335.868327] LustreError: 8866:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 335.871487] Lustre: Skipped 1 previous similar message [ 337.092263] Lustre: server umount lustre-MDT0001 complete [ 340.941483] Lustre: server umount lustre-OST0000 complete [ 344.648317] Lustre: server umount lustre-OST0001 complete [ 351.334775] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 359.897580] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 395.013616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 395.441570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 395.489533] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 395.668101] Lustre: lustre-MDT0000: new disk, initializing [ 395.800275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 395.841769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 401.315277] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 413.478969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.688005] Lustre: 14979:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 413.813286] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 413.818023] Lustre: Skipped 1 previous similar message [ 414.025197] Lustre: lustre-MDT0001: new disk, initializing [ 414.171383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 414.267263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 414.283728] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 419.267553] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 426.109077] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 429.124619] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 435.341449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 435.888572] Lustre: lustre-OST0000: new disk, initializing [ 435.893593] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 435.899923] Lustre: 16719:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 436.048549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 437.078291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 437.095455] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 437.213306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 442.723845] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 450.543166] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 455.829304] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 456.042599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 461.566916] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 461.790937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 467.959890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 467.966677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.989432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 468.002515] Lustre: Skipped 2 previous similar messages [ 468.025330] Lustre: Skipped 1 previous similar message [ 470.461201] Lustre: server umount lustre-OST0000 complete [ 475.618803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 475.637593] LustreError: Skipped 1 previous similar message [ 475.646873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.673372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 480.956245] Lustre: server umount lustre-MDT0000 complete [ 483.302418] LustreError: 14990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 483.327047] LustreError: 14990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 485.134163] LustreError: 16718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780450715 with bad export cookie 12554079948543895444 [ 485.136968] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 485.149029] LustreError: 16718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 485.662897] Lustre: server umount lustre-MDT0001 complete [ 489.954903] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 21:38:38 (1780450718) [ 492.238751] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 493.743843] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 21:38:42 (1780450722) [ 500.080109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.500042] LustreError: 18982:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.572357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 504.646126] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 505.828147] LustreError: 18983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 512.988406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.487759] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 523.170119] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 526.875092] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 533.431687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 539.525096] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 545.445778] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 554.466593] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 554.471115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.479629] Lustre: Skipped 1 previous similar message [ 554.483958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 554.488076] Lustre: Skipped 2 previous similar messages [ 554.880572] Lustre: server umount lustre-OST0000 complete [ 559.592613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 568.895287] Lustre: server umount lustre-MDT0000 complete [ 570.848108] LustreError: 18983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.878920] LustreError: 18983:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 572.414902] LustreError: 18962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780450802 with bad export cookie 12554079948543896326 [ 572.434821] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 572.748390] Lustre: server umount lustre-MDT0001 complete [ 584.011534] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 587.010431] Key type lgssc unregistered [ 587.374959] LNet: 22158:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 587.398495] LNetError: 22158:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 587.423417] LNet: Removed LNI 192.168.203.154@tcp [ 588.418166] Key type .llcrypt unregistered [ 588.422085] Key type ._llcrypt unregistered [ 614.516922] Key type ._llcrypt registered [ 614.522979] Key type .llcrypt registered [ 614.677731] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 616.159447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 616.209586] alg: No test for adler32 (adler32-zlib) [ 617.338289] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 617.618067] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 619.352348] Key type lgssc registered [ 620.661408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 629.873255] loop: module loaded [ 629.938939] loop0: detected capacity change from 0 to 204800000 [ 630.220521] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 630.230781] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 630.633984] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 636.512051] loop0: detected capacity change from 0 to 204800000 [ 636.673209] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 636.688688] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 636.701440] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 641.856450] loop0: detected capacity change from 0 to 204800000 [ 648.248781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 648.280519] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 649.834249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 649.930195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 650.137177] Lustre: lustre-MDT0000: new disk, initializing [ 650.297633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 650.319457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 655.400237] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 666.143043] loop1: detected capacity change from 0 to 204800000 [ 672.776644] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 673.044357] Lustre: lustre-OST07c6: new disk, initializing [ 673.052730] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 673.057974] Lustre: Skipped 1 previous similar message [ 673.062696] Lustre: 25309:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 673.140860] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 680.102065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 680.111244] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 680.149633] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 680.150434] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 686.324158] Lustre: 24312:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 686.336049] Lustre: 24312:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 686.346933] Lustre: 24312:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 686.351333] Lustre: 24312:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 686.361963] Lustre: 24312:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 686.370645] Lustre: 24312:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 699.404378] Lustre: 24312:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 699.417632] Lustre: 24312:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 699.427317] Lustre: 24312:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 699.437037] Lustre: 24312:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 699.448866] Lustre: 24312:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 699.458704] Lustre: 24312:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 704.260822] Lustre: 24312:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 704.270083] Lustre: 24312:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 704.278802] Lustre: 24312:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 704.283791] Lustre: 24312:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.290970] Lustre: 24312:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 704.307018] Lustre: 24312:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.322104] Lustre: 24312:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 704.334499] Lustre: 24312:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.346880] Lustre: 24312:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 704.361321] Lustre: 24312:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.373043] Lustre: 24312:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 704.383460] Lustre: 24312:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.619968] Lustre: 24313:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 707.631101] Lustre: 24313:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 707.641256] Lustre: 24313:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 707.651745] Lustre: 24313:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.668483] Lustre: 24313:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 707.677896] Lustre: 24313:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.692286] Lustre: 24313:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 707.702934] Lustre: 24313:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.711743] Lustre: 24313:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 707.716635] Lustre: 24313:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.721220] Lustre: 24313:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 707.726182] Lustre: 24313:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 712.564760] Lustre: 25981:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 712.577584] Lustre: 25981:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 712.588576] Lustre: 25981:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 712.603197] Lustre: 25981:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 712.623983] Lustre: 25981:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 712.639540] Lustre: 25981:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 712.646389] Lustre: 25981:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 712.653588] Lustre: 25981:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 712.659967] Lustre: 25981:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 712.665384] Lustre: 25981:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 712.670918] Lustre: 25981:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 712.675956] Lustre: 25981:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 724.217851] Lustre: 24312:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 724.236770] Lustre: 24312:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 724.245607] Lustre: 24312:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 724.262913] Lustre: 24312:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 724.270277] Lustre: 24312:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 724.278955] Lustre: 24312:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 724.289368] Lustre: 24312:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 724.309154] Lustre: 24312:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 724.322853] Lustre: 24312:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 724.334857] Lustre: 24312:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 724.345799] Lustre: 24312:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 724.357137] Lustre: 24312:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 772.073157] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 772.096842] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 777.193639] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 777.725975] Lustre: server umount lustre-OST07c6 complete [ 786.914311] Lustre: server umount lustre-MDT0000 complete [ 802.366954] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 805.051829] Key type lgssc unregistered [ 805.343036] LNet: 27828:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.353164] LNetError: 27828:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.404139] LNet: Removed LNI 192.168.203.154@tcp [ 806.150736] Key type .llcrypt unregistered [ 806.152668] Key type ._llcrypt unregistered [ 825.186592] Key type ._llcrypt registered [ 825.189848] Key type .llcrypt registered [ 825.350700] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 826.936696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 826.978032] alg: No test for adler32 (adler32-zlib) [ 828.283905] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 828.555166] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 830.311154] Key type lgssc registered [ 831.831878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 841.219762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 841.291502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 843.068333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 846.933207] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 852.457972] LustreError: 29083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 852.488244] LustreError: 29083:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 857.568962] LustreError: 29084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 858.414561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.809519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 864.729622] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 869.983202] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 874.174425] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 880.924647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 881.372190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 887.559276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 887.626934] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 893.337655] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 908.272541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 908.279385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 908.286147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 908.315235] Lustre: Skipped 1 previous similar message [ 908.332679] Lustre: Skipped 1 previous similar message [ 913.377510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 913.389347] Lustre: Skipped 1 previous similar message [ 913.715996] Lustre: server umount lustre-OST0000 complete [ 918.498197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 918.501699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 918.522323] Lustre: Skipped 1 previous similar message [ 918.537491] Lustre: Skipped 1 previous similar message [ 923.357624] Lustre: server umount lustre-MDT0000 complete [ 923.620374] LustreError: 29084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 923.645073] LustreError: 29084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 927.035989] LustreError: 29063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780451157 with bad export cookie 18271873031978463315 [ 927.051550] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 927.478323] Lustre: server umount lustre-MDT0001 complete [ 936.882286] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 939.513533] Key type lgssc unregistered [ 939.869108] LNet: 32465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.890457] LNetError: 32465:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 939.908428] LNet: Removed LNI 192.168.203.154@tcp [ 941.140088] Key type .llcrypt unregistered [ 941.143944] Key type ._llcrypt unregistered [ 963.620357] Key type ._llcrypt registered [ 963.624909] Key type .llcrypt registered [ 963.725948] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 964.633974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 964.721140] alg: No test for adler32 (adler32-zlib) [ 965.858716] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 966.139853] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 967.799289] Key type lgssc registered [ 969.348419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 978.001573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 978.038391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 979.524780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 983.786661] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 989.667994] LustreError: 33720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 989.698618] LustreError: 33720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 992.388384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.718423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 997.313335] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1003.563839] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1008.186504] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1014.777688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1015.052431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1021.281037] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1025.342980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1027.669825] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1037.792432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1037.800498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1037.829558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1040.867986] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1040.881901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1040.884921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1040.911488] Lustre: Skipped 1 previous similar message [ 1043.982765] Lustre: server umount lustre-OST0000 complete [ 1049.056528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1049.070939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1049.092455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1051.105420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1051.106356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1053.772101] Lustre: server umount lustre-MDT0000 complete [ 1056.226812] LustreError: 33720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1056.258941] LustreError: 33720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1057.050326] LustreError: 33699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780451287 with bad export cookie 1995174969736913012 [ 1057.059828] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1057.065313] LustreError: 33699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1057.275531] Lustre: server umount lustre-MDT0001 complete [ 1068.163968] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1071.940518] Key type lgssc unregistered [ 1072.316887] LNet: 36898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1072.344374] LNetError: 36898:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1072.386789] LNet: Removed LNI 192.168.203.154@tcp [ 1073.445218] Key type .llcrypt unregistered [ 1073.450146] Key type ._llcrypt unregistered [ 1097.259522] Key type ._llcrypt registered [ 1097.264033] Key type .llcrypt registered [ 1097.381942] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1098.194734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1098.211987] alg: No test for adler32 (adler32-zlib) [ 1099.249374] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 1099.468802] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1101.127180] Key type lgssc registered [ 1102.393291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1109.760873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1109.779537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1111.255979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.484214] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1121.761143] LustreError: 38300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1121.779395] LustreError: 38300:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1125.819951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1126.256787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1131.103166] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1137.447206] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1142.005729] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1148.954783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1149.539734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1156.029180] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1156.742951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1161.449708] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1167.328448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1167.335060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1167.347722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1167.376692] Lustre: Skipped 2 previous similar messages [ 1168.155364] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.54@tcp (stopping) [ 1168.161860] Lustre: Skipped 2 previous similar messages [ 1169.230879] Lustre: server umount lustre-MDT0000 complete [ 1172.454356] LustreError: 38295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1172.479270] LustreError: 38295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1173.016263] LustreError: 38280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780451403 with bad export cookie 7188067587877340052 [ 1173.029121] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1173.031777] LustreError: 38280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1173.264920] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.54@tcp (stopping) [ 1177.572451] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1177.597681] LustreError: 38297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1177.600472] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1177.622219] LustreError: 38297:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1179.495255] Lustre: server umount lustre-MDT0001 complete [ 1209.311162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1209.630341] Lustre: server umount lustre-OST0000 complete [ 1226.946804] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1229.890767] Key type lgssc unregistered [ 1230.189176] LNet: 41806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1230.195887] LNetError: 41806:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1230.213785] LNet: Removed LNI 192.168.203.154@tcp [ 1231.020850] Key type .llcrypt unregistered [ 1231.024566] Key type ._llcrypt unregistered [ 1252.981195] Key type ._llcrypt registered [ 1252.983970] Key type .llcrypt registered [ 1253.111283] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1254.264936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1254.408348] alg: No test for adler32 (adler32-zlib) [ 1255.472678] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 1255.759791] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1257.399173] Key type lgssc registered [ 1258.574427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1266.484233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1266.517613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1267.986836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1272.451911] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1277.415507] LustreError: 43205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1277.443687] LustreError: 43205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1281.123191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.587197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1286.043915] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1291.711741] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1295.372451] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1301.881891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1302.192498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1308.226900] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1312.434845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1314.077772] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1321.229248] Lustre: lustre-OST0000: Not available for connect from 192.168.203.54@tcp (stopping) [ 1321.953630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1321.962735] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1321.977844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1322.976866] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1322.979631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1322.988504] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.017323] Lustre: Skipped 1 previous similar message [ 1326.054292] Lustre: server umount lustre-OST0000 complete [ 1348.064854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1348.078267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1348.100512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1354.721297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1354.735862] Lustre: Skipped 4 previous similar messages [ 1358.303307] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1358.634428] Lustre: server umount lustre-MDT0000 complete [ 1359.842789] LustreError: 43202:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1359.876742] LustreError: 43202:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1362.977612] LustreError: 43185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780451592 with bad export cookie 12689056414754642742 [ 1362.988511] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1363.005190] LustreError: 43185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1369.538619] Lustre: server umount lustre-MDT0001 complete [ 1380.581962] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1384.271870] Key type lgssc unregistered [ 1384.761826] LNet: 46557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1384.783772] LNetError: 46557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1384.809675] LNet: Removed LNI 192.168.203.154@tcp [ 1385.851215] Key type .llcrypt unregistered [ 1385.853079] Key type ._llcrypt unregistered [ 1410.863795] Key type ._llcrypt registered [ 1410.866598] Key type .llcrypt registered [ 1410.985727] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1412.062457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1412.130268] alg: No test for adler32 (adler32-zlib) [ 1413.208547] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 1413.457563] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1415.103230] Key type lgssc registered [ 1416.220745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1424.831391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1424.866786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1426.437725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1431.300600] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1436.642332] LustreError: 47960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1436.671836] LustreError: 47960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1437.667233] LustreError: 47959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1439.795856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1440.207079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1444.728932] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1451.265409] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1455.538572] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1462.568231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1463.028077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1469.138736] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1472.304541] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1475.518635] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1481.846917] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1485.907082] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1498.659808] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-0221h55m26s [ 1500.250129] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1503.200991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1503.214485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1504.001550] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.54@tcp (stopping) [ 1504.016216] Lustre: Skipped 2 previous similar messages [ 1505.076797] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.54@tcp (stopping) [ 1505.086384] Lustre: Skipped 3 previous similar messages [ 1508.918027] Lustre: server umount lustre-MDT0000 complete [ 1510.122677] LustreError: 47955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1512.421485] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1516.283381] LustreError: 48936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1516.306600] LustreError: 48936:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1520.581314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1520.818206] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1521.202814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1521.305536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1522.269175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1527.274383] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1531.095548] Lustre: DEBUG MARKER: Wait for df (36077) ... [ 1532.775422] Lustre: DEBUG MARKER: done [ 1541.602120] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1541.623807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1541.649340] Lustre: Skipped 2 previous similar messages [ 1541.656341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1542.123975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1543.764498] Lustre: server umount lustre-OST0000 complete [ 1548.260514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1548.265361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1548.279888] Lustre: Skipped 1 previous similar message [ 1548.288277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1548.291460] Lustre: Skipped 2 previous similar messages [ 1554.531508] Lustre: server umount lustre-MDT0000 complete [ 1559.023688] LustreError: 47940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780451789 with bad export cookie 17781614848642197895 [ 1559.036805] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1559.038179] LustreError: 47940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1559.568113] Lustre: server umount lustre-MDT0001 complete [ 1574.710727] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1578.919502] Key type lgssc unregistered [ 1579.380237] LNet: 52513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1579.389138] LNetError: 52513:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1579.427789] LNet: Removed LNI 192.168.203.154@tcp [ 1580.213253] Key type .llcrypt unregistered [ 1580.216407] Key type ._llcrypt unregistered [ 1628.949218] Key type ._llcrypt registered [ 1628.953118] Key type .llcrypt registered [ 1629.078403] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1630.753241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1630.799344] alg: No test for adler32 (adler32-zlib) [ 1632.108933] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 1632.527850] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1634.271176] Key type lgssc registered [ 1635.529781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1646.921772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1646.936779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1648.397686] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1648.420561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1648.628688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1654.844377] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1665.496644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1665.617093] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1665.662187] Lustre: 54707:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1665.996642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1671.121957] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1678.351778] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1682.427215] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1690.880240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1691.257159] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1691.634577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1697.499438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1699.781250] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1706.549782] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1712.702218] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-0221h59m00s [ 1714.279630] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1721.361652] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1724.879689] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1725.103105] LustreError: 54715:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1745.170278] Lustre: lustre-MDT0000: Client 98816ee1-1a63-4f42-a519-d65a1127300f (at 192.168.203.54@tcp) reconnecting [ 1765.127112] LustreError: 54715:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1767.576239] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1770.287458] Lustre: DEBUG MARKER: done [ 1779.169390] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1779.174781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1779.184481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1779.680900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1779.698320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1779.720534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1782.757780] Lustre: server umount lustre-OST0000 complete [ 1788.907612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1788.914523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1788.933231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1792.242389] Lustre: server umount lustre-MDT0000 complete [ 1795.726341] LustreError: 54697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780452025 with bad export cookie 14219105944819031131 [ 1795.733696] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1795.742410] LustreError: 54697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1796.072753] Lustre: server umount lustre-MDT0001 complete [ 1805.761234] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1808.438351] Key type lgssc unregistered [ 1808.778186] LNet: 58680:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1808.788772] LNetError: 58680:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1808.808282] LNet: Removed LNI 192.168.203.154@tcp [ 1809.572890] Key type .llcrypt unregistered [ 1809.580205] Key type ._llcrypt unregistered [ 1854.378901] Key type ._llcrypt registered [ 1854.382425] Key type .llcrypt registered [ 1854.498243] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1855.525159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1855.541209] alg: No test for adler32 (adler32-zlib) [ 1856.629493] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 1856.951849] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1858.695403] Key type lgssc registered [ 1859.975861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1863.892354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1863.929347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1865.413949] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1865.455151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1865.754949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1866.751328] Lustre: Failing over lustre-MDT0000 [ 1867.098750] Lustre: server umount lustre-MDT0000 complete [ 1875.392452] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 22:01:43 (1780452103) [ 1884.239978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1884.835606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1889.604270] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1898.261800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1898.417398] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1898.466847] Lustre: 61477:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1898.800765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1903.636724] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1910.734773] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1914.453775] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1921.656083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1921.813452] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1922.017244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.875847] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1931.934241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:291 to 0x280000401:321) [ 1934.881670] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1941.425484] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1950.175973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1950.182415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1950.203249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1952.741182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1952.743784] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1955.407832] Lustre: server umount lustre-MDT0000 complete [ 1959.895120] LustreError: 61468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780452189 with bad export cookie 4017712546344706052 [ 1959.913718] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1960.295808] Lustre: server umount lustre-MDT0001 complete [ 1964.140220] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1969.743890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1978.275675] Lustre: 60398:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780452192/real 1780452192] req@0000000060391fb9 x1866939326570112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780452208 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1978.309757] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1978.320284] Lustre: Skipped 1 previous similar message [ 1979.914195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1981.407172] Lustre: 60400:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780452195/real 1780452195] req@00000000a89189b9 x1866939326570240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780452211 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1983.455171] Lustre: 60399:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780452197/real 1780452197] req@000000007dc05f76 x1866939326570368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780452213 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1984.484311] LustreError: 60396:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cb08b71b x1866939326571136/t0(0) o250->MGC192.168.203.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1984.880890] LustreError: 63098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1984.905795] LustreError: 63098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1985.006534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1985.130040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1985.503192] Lustre: 60398:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780452199/real 1780452199] req@00000000dfec02f2 x1866939326570624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780452215 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1988.003394] LustreError: 63099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1989.039678] LustreError: 63098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1989.076378] LustreError: 63098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1990.050772] Lustre: 60397:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780452204/real 1780452204] req@0000000006fad348 x1866939326570752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780452220 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1990.107639] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1994.216268] LustreError: 64907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1994.248255] LustreError: 64907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1999.335915] LustreError: 64908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1999.363882] LustreError: 64908:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2001.044188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2001.577045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2007.027257] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2007.211709] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2008.114898] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2014.635223] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2020.136424] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2032.099980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2032.110705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2032.149525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2032.168730] Lustre: Skipped 2 previous similar messages [ 2038.763145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2038.773232] Lustre: Skipped 4 previous similar messages [ 2043.363332] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2043.850561] Lustre: server umount lustre-MDT0000 complete [ 2043.886581] LustreError: 64909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2043.906596] LustreError: 64909:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2048.070341] LustreError: 61469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780452278 with bad export cookie 4017712546344708327 [ 2048.086910] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2048.091101] LustreError: 61469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2048.682497] Lustre: server umount lustre-MDT0001 complete [ 2056.244488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2064.351210] Lustre: 60399:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780452278/real 1780452278] req@0000000061a33630 x1866939326600576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780452294 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2064.409333] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.424719] Lustre: Skipped 2 previous similar messages [ 2067.111233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.637051] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x37c1c8462e81ae4a [ 2073.667168] Lustre: MGC192.168.203.154@tcp: Connection restored to 0@lo (at 0@lo) [ 2074.072807] LustreError: 65842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2074.216794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2074.295032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2079.255927] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2089.425689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2095.101877] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2095.626734] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2103.003858] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2107.687195] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2118.627105] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2118.649854] Lustre: Skipped 1 previous similar message [ 2118.653467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2118.685104] Lustre: Skipped 2 previous similar messages [ 2128.867190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2128.875366] Lustre: Skipped 3 previous similar messages [ 2130.399201] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2130.700137] Lustre: server umount lustre-MDT0000 complete [ 2133.986377] LustreError: 67370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2134.007396] LustreError: 67370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 2134.990547] LustreError: 64049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780452364 with bad export cookie 4017712546344709706 [ 2135.000780] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2135.001990] LustreError: 64049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2135.477049] Lustre: server umount lustre-MDT0001 complete [ 2143.349020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2147.876722] Lustre: DEBUG MARKER: files compared the same [ 2161.052757] Lustre: server umount lustre-OST0000 complete [ 2180.885686] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2184.730409] Key type lgssc unregistered [ 2185.018749] LNet: 70403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2185.035965] LNetError: 70403:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2185.089330] LNet: Removed LNI 192.168.203.154@tcp [ 2186.293587] Key type .llcrypt unregistered [ 2186.297511] Key type ._llcrypt unregistered [ 2212.419195] Key type ._llcrypt registered [ 2212.421655] Key type .llcrypt registered [ 2212.576536] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2213.890685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2213.916668] alg: No test for adler32 (adler32-zlib) [ 2215.184899] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 2215.568422] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 2217.377278] Key type lgssc registered [ 2218.998909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2229.273826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2229.329393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2230.900174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2237.059829] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug +malloc all [ 2240.492416] LustreError: 71805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2240.524260] LustreError: 71805:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2245.608265] LustreError: 71806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2247.463133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2247.891570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2252.661134] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug +malloc all [ 2259.251929] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2263.853508] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2272.386603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2272.675170] LustreError: 73400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2272.861114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2275.958917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2280.712534] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug +malloc all [ 2289.168923] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2301.925287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2301.937126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2301.940605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2301.954593] Lustre: Skipped 1 previous similar message [ 2305.609752] Lustre: server umount lustre-OST0000 complete [ 2312.163347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2312.164363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2312.181514] Lustre: Skipped 1 previous similar message [ 2312.198050] Lustre: Skipped 2 previous similar messages [ 2317.259737] Lustre: server umount lustre-MDT0000 complete [ 2317.286853] LustreError: 71805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2317.354328] LustreError: 71805:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2322.010307] LustreError: 71786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780452552 with bad export cookie 15488135128186355788 [ 2322.011928] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2322.028130] LustreError: 71786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2322.491109] Lustre: server umount lustre-MDT0001 complete [ 2334.866912] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2338.150359] Key type lgssc unregistered [ 2338.504546] LNet: 74983:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2338.508630] LNetError: 74983:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2338.547223] LNet: Removed LNI 192.168.203.154@tcp [ 2339.656808] Key type .llcrypt unregistered [ 2339.664205] Key type ._llcrypt unregistered [ 2365.618697] Key type ._llcrypt registered [ 2365.622561] Key type .llcrypt registered [ 2365.758924] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2367.031387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2367.211415] alg: No test for adler32 (adler32-zlib) [ 2368.458294] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 2368.781878] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 2370.545726] Key type lgssc registered [ 2372.055701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2381.987868] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2382.026298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2397.791365] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2402.975510] LustreError: 76358:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.154@tcp: failed processing log, type 4: rc = -110 [ 2429.791325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2436.894565] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2443.716958] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2452.350383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2452.570045] Lustre: *** cfs_fail_loc=706, val=0*** [ 2453.053313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2453.136946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2457.974368] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2458.083540] LustreError: 76383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.105447] LustreError: 76383:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2463.209415] LustreError: 76384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2464.243792] LustreError: 77332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2464.274747] LustreError: 77332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2468.146683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2468.665552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2474.577428] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2481.387105] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2485.953692] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2488.805034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2488.821077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2488.846565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2489.338313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2489.356040] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2489.383632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2489.396056] Lustre: Skipped 1 previous similar message [ 2494.439927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2494.726940] Lustre: server umount lustre-OST0000 complete [ 2499.567921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2499.571293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2499.571949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2499.571954] Lustre: Skipped 1 previous similar message [ 2499.594582] Lustre: Skipped 1 previous similar message [ 2504.674527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2504.684653] Lustre: Skipped 2 previous similar messages [ 2505.688738] Lustre: server umount lustre-MDT0000 complete [ 2509.794161] LustreError: 77337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2509.823431] LustreError: 77337:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2510.220725] LustreError: 78057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780452740 with bad export cookie 1354745252306119245 [ 2510.234797] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2510.237093] LustreError: 78057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2510.818457] Lustre: server umount lustre-MDT0001 complete [ 2522.221815] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2525.730188] Key type lgssc unregistered [ 2526.121712] LNet: 79620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2526.136282] LNetError: 79620:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2526.156872] LNet: Removed LNI 192.168.203.154@tcp [ 2527.236165] Key type .llcrypt unregistered [ 2527.240758] Key type ._llcrypt unregistered [ 2555.115540] Key type ._llcrypt registered [ 2555.126780] Key type .llcrypt registered [ 2555.306560] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2556.895041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2557.093935] alg: No test for adler32 (adler32-zlib) [ 2558.204719] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 2558.567640] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 2560.391374] Key type lgssc registered [ 2561.880521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2571.403444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2571.429307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2577.798894] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2586.422926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2587.091393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2592.240866] LustreError: 81693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2592.263413] LustreError: 81693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2592.395546] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2596.321578] LustreError: 81696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.347228] LustreError: 81696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2601.440489] LustreError: 81697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.471154] LustreError: 81697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2601.780044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2602.164279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2607.586366] LustreError: 81696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2607.610573] LustreError: 81696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2609.869650] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2611.690232] LustreError: 81697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.723859] LustreError: 81697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2619.612362] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2619.626345] Lustre: MGS: Client 955d1063-e9f8-4150-ad8e-6101a4a35ee8 (at 0@lo) reconnecting [ 2619.649130] LustreError: 81707:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2619.937704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2620.003622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2625.412257] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2641.376551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2641.388871] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2641.416416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2643.378958] Lustre: server umount lustre-OST0000 complete [ 2650.594590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2650.595337] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2650.607278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2650.630570] Lustre: Skipped 1 previous similar message [ 2650.650168] Lustre: Skipped 1 previous similar message [ 2655.714705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2655.724466] Lustre: Skipped 1 previous similar message [ 2657.826738] Lustre: server umount lustre-MDT0000 complete [ 2660.837087] LustreError: 81696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2660.860499] LustreError: 81696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2662.412678] Lustre: server umount lustre-MDT0001 complete [ 2666.521354] Lustre: server umount MGS complete [ 2679.582935] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 22:15:08 (1780452908) [ 2715.782551] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 2727.732313] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2770.431920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2775.067261] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2785.719776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.079338] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2786.157867] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2786.256660] Lustre: lustre-MDT0001: new disk, initializing [ 2786.404832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2786.439312] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2786.449374] Lustre: 88785:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2786.461493] LustreError: 88785:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2791.486537] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2803.291735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2803.564684] Lustre: lustre-OST0000: new disk, initializing [ 2803.576093] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2803.592181] Lustre: 89535:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2803.706626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2810.373899] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2812.578054] Lustre: 89914:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000ea243e6e x1866940062672128/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1780453058 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2818.784644] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2826.978102] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2827.007247] Lustre: MGS: Client 3479462b-d89d-483f-bd11-ea32536dc993 (at 0@lo) reconnecting [ 2827.077753] Lustre: 90468:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2827.108091] LustreError: 88771:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2827.124084] LustreError: 88771:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2827.139583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2827.254694] Lustre: lustre-MDT0000: new disk, initializing [ 2827.414463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2827.440709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2827.466433] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2828.513121] Lustre: 90492:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000d91aa13c x1866940062678784/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1780453074 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2832.441824] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2833.695807] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2837.616158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2837.630356] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2837.785941] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2863.595253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2863.603605] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2863.620565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2869.992565] Lustre: server umount lustre-OST0000 complete [ 2875.361034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2875.366856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2875.386954] Lustre: Skipped 1 previous similar message [ 2875.391695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2875.391704] Lustre: Skipped 2 previous similar messages [ 2880.688768] Lustre: server umount lustre-MDT0000 complete [ 2890.971747] Lustre: server umount lustre-MDT0001 complete [ 2894.696630] Lustre: server umount MGS complete [ 2906.386704] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 22:18:54 (1780453134) [ 2924.023492] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2926.997315] Key type lgssc unregistered [ 2927.324733] LNet: 92956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2927.335069] LNetError: 92956:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2927.383438] LNet: Removed LNI 192.168.203.154@tcp [ 2928.151860] Key type .llcrypt unregistered [ 2928.153661] Key type ._llcrypt unregistered [ 2950.785287] Key type ._llcrypt registered [ 2950.793219] Key type .llcrypt registered [ 2951.013107] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2953.409044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2953.455756] alg: No test for adler32 (adler32-zlib) [ 2954.953929] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 2955.480480] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 2957.329105] Key type lgssc registered [ 2958.955902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2969.423883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2969.441680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2970.967903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2975.766498] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2980.331369] LustreError: 94159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2980.364422] LustreError: 94159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2985.441784] LustreError: 94160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2986.115760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2986.449317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2990.831071] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2997.178334] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3001.136602] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3008.709457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3009.260634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3015.480151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3015.805262] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3023.385648] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3036.130584] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3036.141844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3036.153236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3036.189848] Lustre: Skipped 1 previous similar message [ 3038.752579] Lustre: server umount lustre-OST0000 complete [ 3042.797384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3042.814185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3042.840892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3042.845738] Lustre: Skipped 1 previous similar message [ 3046.370945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3046.373206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3046.408131] Lustre: Skipped 1 previous similar message [ 3048.998993] Lustre: server umount lustre-MDT0000 complete [ 3051.490081] LustreError: 95270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3051.509857] LustreError: 95270:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3052.732231] LustreError: 95776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780453282 with bad export cookie 4523295910808568723 [ 3052.737412] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3052.743240] LustreError: 95776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3053.263127] Lustre: server umount lustre-MDT0001 complete [ 3062.217370] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3064.697882] Key type lgssc unregistered [ 3064.975895] LNet: 97337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3064.988910] LNetError: 97337:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3065.014704] LNet: Removed LNI 192.168.203.154@tcp [ 3065.831695] Key type .llcrypt unregistered [ 3065.834443] Key type ._llcrypt unregistered [ 3080.597957] Key type ._llcrypt registered [ 3080.600623] Key type .llcrypt registered [ 3080.735946] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3082.017936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3082.075754] alg: No test for adler32 (adler32-zlib) [ 3083.105408] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 3083.353277] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3085.063163] Key type lgssc registered [ 3086.082408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3092.459182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3092.462341] LustreError: 98404:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3092.473488] LustreError: 98403:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3092.479133] LustreError: 98404:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4992 [ 3092.487699] LustreError: 98404:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3092.493390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3092.494609] LustreError: 98404:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3094.042412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3102.746597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3103.185277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3108.287511] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3116.332966] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing lsmod [ 3120.457868] LustreError: 99572:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3120.494537] LustreError: 99581:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3120.499399] LustreError: 99572:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4978 [ 3120.521136] LustreError: 99572:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3120.530804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3120.544230] LustreError: 99572:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3121.354280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3126.753822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3126.757532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3126.783604] Lustre: Skipped 1 previous similar message [ 3126.808384] Lustre: Skipped 2 previous similar messages [ 3128.865649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3128.868762] Lustre: Skipped 2 previous similar messages [ 3131.763831] Lustre: server umount lustre-MDT0000 complete [ 3133.921798] LustreError: 98437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3133.968851] LustreError: 98437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3136.724637] LustreError: 98417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780453366 with bad export cookie 14205901060773482989 [ 3136.737740] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3136.741990] LustreError: 98417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3137.087351] LustreError: 98874:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007994eb08 x1866940613154816/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3137.387737] Lustre: server umount lustre-MDT0001 complete [ 3151.631513] Lustre: server umount lustre-OST0000 complete [ 3163.529837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3164.006936] LustreError: 100645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3164.083184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3168.455703] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3169.249705] LustreError: 100646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3174.381040] LustreError: 100645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3178.253827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3178.752329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3183.650375] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3189.651452] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3194.436529] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3201.456290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3201.824022] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3208.671702] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3210.041598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3214.545287] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3224.545339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3224.559887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3224.580626] Lustre: Skipped 1 previous similar message [ 3224.585258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3225.582658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3225.593930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3228.726533] Lustre: server umount lustre-OST0000 complete [ 3235.296340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3235.306193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3235.324423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3235.336804] Lustre: Skipped 2 previous similar messages [ 3238.739182] Lustre: server umount lustre-MDT0000 complete [ 3240.938201] LustreError: 100642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3240.984076] LustreError: 100642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3243.053585] LustreError: 100628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780453473 with bad export cookie 14205901060773483689 [ 3243.067194] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3243.389703] Lustre: server umount lustre-MDT0001 complete [ 3255.929419] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3258.998698] Key type lgssc unregistered [ 3259.377254] LNet: 103822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3259.396304] LNetError: 103822:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3259.441496] LNet: Removed LNI 192.168.203.154@tcp [ 3260.554085] Key type .llcrypt unregistered [ 3260.559363] Key type ._llcrypt unregistered [ 3285.229630] Key type ._llcrypt registered [ 3285.234618] Key type .llcrypt registered [ 3285.378265] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3286.660252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3286.841704] alg: No test for adler32 (adler32-zlib) [ 3288.114149] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 3288.388557] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3290.063534] Key type lgssc registered [ 3291.477354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3300.162403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3300.195933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3301.876155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3306.482430] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3311.138651] LustreError: 105223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3311.158123] LustreError: 105223:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3315.485431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3315.974687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3321.581611] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3328.542573] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3333.232936] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3340.952559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3341.402713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3346.577397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3349.250325] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3355.864196] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3364.682669] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3368.532917] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3376.609382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3376.640714] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3376.641354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3376.685397] Lustre: Skipped 1 previous similar message [ 3381.733520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3381.748480] Lustre: Skipped 1 previous similar message [ 3386.852317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3386.862937] Lustre: Skipped 1 previous similar message [ 3387.359234] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3387.530119] Lustre: server umount lustre-OST0000 complete [ 3392.993253] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3392.999193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3393.023366] Lustre: Skipped 1 previous similar message [ 3393.027907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3397.089342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3397.090072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3398.786101] Lustre: server umount lustre-MDT0000 complete [ 3402.209266] LustreError: 105224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3402.238113] LustreError: 105224:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3402.737740] LustreError: 105205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780453632 with bad export cookie 15919100736108714627 [ 3402.750206] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3402.764269] LustreError: 105205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3403.225918] Lustre: server umount lustre-MDT0001 complete [ 3414.278453] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3417.352066] Key type lgssc unregistered [ 3417.739569] LNet: 108507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3417.751798] LNetError: 108507:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3417.780288] LNet: Removed LNI 192.168.203.154@tcp [ 3418.645286] Key type .llcrypt unregistered [ 3418.648313] Key type ._llcrypt unregistered [ 3435.381575] Key type ._llcrypt registered [ 3435.383816] Key type .llcrypt registered [ 3435.505297] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3436.353730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3436.477132] alg: No test for adler32 (adler32-zlib) [ 3437.522776] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 3437.701817] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3439.359230] Key type lgssc registered [ 3440.388337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3447.458957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3447.483679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3448.867910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3453.935986] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3458.031065] LustreError: 109714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3458.054809] LustreError: 109714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3463.054708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3463.147525] LustreError: 109715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3463.477987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3468.930784] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3475.305799] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3479.448617] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3486.678474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3487.233395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3493.438959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3494.094620] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3501.191738] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3513.831772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3513.849725] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3513.860656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3513.881419] LustreError: Skipped 1 previous similar message [ 3513.893283] Lustre: Skipped 1 previous similar message [ 3518.166257] Lustre: server umount lustre-OST0000 complete [ 3524.065106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3524.073720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3524.077318] Lustre: Skipped 2 previous similar messages [ 3524.091537] Lustre: Skipped 1 previous similar message [ 3529.185471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3529.612735] Lustre: server umount lustre-MDT0000 complete [ 3533.283823] LustreError: 109715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3533.307528] LustreError: 109715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3533.941504] LustreError: 109695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780453763 with bad export cookie 9703104044415088566 [ 3533.949613] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3533.954579] LustreError: 109695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3534.611142] Lustre: server umount lustre-MDT0001 complete [ 3548.004893] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3551.089528] Key type lgssc unregistered [ 3551.412587] LNet: 112901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3551.423694] LNetError: 112901:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3552.493304] LNet: Removed LNI 192.168.203.154@tcp [ 3553.340527] Key type .llcrypt unregistered [ 3553.345107] Key type ._llcrypt unregistered [ 3577.800496] Key type ._llcrypt registered [ 3577.803408] Key type .llcrypt registered [ 3577.933224] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3579.215807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3579.328109] alg: No test for adler32 (adler32-zlib) [ 3580.461607] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 3580.730060] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3582.391169] Key type lgssc registered [ 3583.584510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3592.012555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3592.054970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3593.582131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3598.003503] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3603.936359] LustreError: 114303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3603.970169] LustreError: 114303:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3606.874790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3607.184522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3611.663152] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3617.911309] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3622.043637] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3630.294325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3630.700049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3637.078597] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3638.902896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3644.448407] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3651.902077] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3655.524609] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3655.531649] Lustre: Skipped 2 previous similar messages [ 3658.389829] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3658.429026] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3660.149247] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3660.153637] Lustre: Skipped 2 previous similar messages [ 3667.527879] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3667.536517] Lustre: Skipped 1 previous similar message [ 3672.468382] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.54@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3672.475535] Lustre: Skipped 2 previous similar messages [ 3676.649817] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.54@tcp 192.168.0.[2,10]@tcp [ 3685.290204] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.54/24@tcp in log lustre-MDT0000 [ 3685.302307] Lustre: Skipped 2 previous similar messages [ 3695.253131] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.54/24@tcp [ 3695.258373] Lustre: Skipped 1 previous similar message [ 3704.803130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3704.820734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3704.843920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3705.826629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3705.830680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3708.598516] Lustre: server umount lustre-OST0000 complete [ 3714.539157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3714.551347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3714.576193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3714.587185] Lustre: Skipped 1 previous similar message [ 3719.150253] Lustre: server umount lustre-MDT0000 complete [ 3721.185705] LustreError: 115462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3721.220864] LustreError: 115462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3723.997468] LustreError: 115021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780453953 with bad export cookie 18370315302591054878 [ 3724.000245] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3724.015032] LustreError: 115021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3724.641338] Lustre: server umount lustre-MDT0001 complete [ 3737.374817] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3741.423895] Key type lgssc unregistered [ 3741.827519] LNet: 119038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3741.837260] LNetError: 119038:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3741.899174] LNet: Removed LNI 192.168.203.154@tcp [ 3743.091394] Key type .llcrypt unregistered [ 3743.096939] Key type ._llcrypt unregistered [ 3774.734602] Key type ._llcrypt registered [ 3774.741920] Key type .llcrypt registered [ 3774.894090] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3776.273125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3776.438278] alg: No test for adler32 (adler32-zlib) [ 3777.567867] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 3777.887266] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3779.647249] Key type lgssc registered [ 3781.098286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3789.068473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3789.100190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3790.632295] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3790.667721] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.54/24@tcp [ 3790.678731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3795.345123] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3800.035709] LustreError: 120581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3800.051142] LustreError: 120581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3804.609680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3804.962573] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3805.021883] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.54/24@tcp [ 3805.044333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3809.538797] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3816.246709] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3820.417385] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3828.176557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3828.503071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3834.272448] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3835.766144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3841.160517] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3855.329025] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3855.341851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3855.363228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3856.372400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3856.374534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3856.399370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3858.480578] Lustre: server umount lustre-OST0000 complete [ 3866.610827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3866.611536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3866.626812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3866.626820] Lustre: Skipped 1 previous similar message [ 3866.631177] Lustre: Skipped 1 previous similar message [ 3871.713180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3871.720943] Lustre: Skipped 2 previous similar messages [ 3876.833621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3876.840557] Lustre: Skipped 1 previous similar message [ 3877.343128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3877.770572] Lustre: server umount lustre-MDT0000 complete [ 3881.955468] LustreError: 120581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3881.986793] LustreError: 120581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3882.034614] LustreError: 120561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780454112 with bad export cookie 1460112712621372230 [ 3882.047150] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3882.063283] LustreError: 120561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3882.625959] Lustre: server umount lustre-MDT0001 complete [ 3893.906874] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3897.001977] Key type lgssc unregistered [ 3897.402930] LNet: 123815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3897.412133] LNetError: 123815:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3897.430786] LNet: Removed LNI 192.168.203.154@tcp [ 3898.359600] Key type .llcrypt unregistered [ 3898.361639] Key type ._llcrypt unregistered [ 3929.296378] Key type ._llcrypt registered [ 3929.298652] Key type .llcrypt registered [ 3929.449420] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 3944.880460] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3946.060780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3946.201153] alg: No test for adler32 (adler32-zlib) [ 3947.324271] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 3947.663350] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3949.367818] Key type lgssc registered [ 3950.580324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3987.699900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3987.787825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3989.108111] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3989.155183] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3989.289620] Lustre: lustre-MDT0000: new disk, initializing [ 3989.444839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3989.474455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3994.739422] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4006.842985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4006.983314] Lustre: 127183:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4007.072216] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4007.080076] Lustre: Skipped 1 previous similar message [ 4007.241978] Lustre: lustre-MDT0001: new disk, initializing [ 4007.374277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4007.438974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4007.463032] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4012.349142] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4020.084981] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4023.812846] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4031.380418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4031.824858] Lustre: lustre-OST0000: new disk, initializing [ 4031.829762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4031.837554] Lustre: 128927:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4032.014047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4038.195917] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4038.723582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4038.734476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4038.853402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4045.675810] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4049.683585] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4049.875774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4054.005436] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4054.219728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4059.633110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4059.639186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4059.665377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4064.755457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4064.764731] Lustre: Skipped 2 previous similar messages [ 4069.870773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4069.874473] Lustre: Skipped 1 previous similar message [ 4070.367064] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4070.620680] Lustre: server umount lustre-OST0000 complete [ 4074.980242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4074.983282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4074.999796] Lustre: Skipped 2 previous similar messages [ 4075.014774] Lustre: Skipped 1 previous similar message [ 4080.097281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4080.109213] Lustre: Skipped 1 previous similar message [ 4080.424045] Lustre: server umount lustre-MDT0000 complete [ 4083.871510] LustreError: 127176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780454313 with bad export cookie 16860727650042599282 [ 4083.879818] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4083.892129] LustreError: 127176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4084.198859] Lustre: server umount lustre-MDT0001 complete [ 4093.284223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4093.816643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4098.372481] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4099.040744] LustreError: 130996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4099.061540] LustreError: 130996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4103.140637] LustreError: 130997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4107.444576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4107.950437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4113.019391] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4119.230958] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4123.245740] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4130.405955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4130.768690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4136.701740] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4143.412687] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4148.429201] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4148.678829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4153.681525] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4153.930976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4162.677634] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4162.847314] Lustre: lustre-OST0001: new disk, initializing [ 4162.851552] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4162.860680] Lustre: 133992:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4162.943311] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4169.860895] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4171.924844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4171.940096] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4172.025268] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4178.685578] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4178.949549] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4183.553288] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4183.798472] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4188.269400] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a24c4ba0800.ost_server_uuid 50 [ 4190.140038] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9a24c4ba0800.ost_server_uuid in FULL state after 0 sec [ 4201.719994] Lustre: server umount lustre-OST0001 complete [ 4202.977242] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4202.989257] LustreError: Skipped 1 previous similar message [ 4202.992097] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4203.015224] LustreError: 132593:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4203.024731] Lustre: Skipped 1 previous similar message [ 4208.098855] LustreError: 134441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4208.108760] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4208.112209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4208.124981] LustreError: 134441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4208.153033] Lustre: Skipped 1 previous similar message [ 4211.995733] Lustre: server umount lustre-OST0000 complete [ 4218.337518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4222.023296] Lustre: server umount lustre-MDT0000 complete [ 4223.457560] LustreError: 130996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4223.486459] LustreError: 130996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4225.828251] LustreError: 130977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780454455 with bad export cookie 16860727650042600164 [ 4225.828251] LustreError: 130978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780454455 with bad export cookie 16860727650042600164 [ 4225.834167] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4225.838640] LustreError: 130978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4226.279957] Lustre: server umount lustre-MDT0001 complete [ 4245.756525] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 4248.514821] Key type lgssc unregistered [ 4248.809261] LNet: 136829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4248.818595] LNetError: 136829:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4248.845162] LNet: Removed LNI 192.168.203.154@tcp [ 4249.666897] Key type .llcrypt unregistered [ 4249.670121] Key type ._llcrypt unregistered [ 4289.477753] Key type ._llcrypt registered [ 4289.480358] Key type .llcrypt registered [ 4289.578340] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 4303.290867] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4304.699326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4304.771576] alg: No test for adler32 (adler32-zlib) [ 4305.839979] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 4306.100444] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 4307.815204] Key type lgssc registered [ 4309.049097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4342.368052] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4342.393953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4343.706766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4343.736626] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4343.838509] Lustre: lustre-MDT0000: new disk, initializing [ 4343.928545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4343.955487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4347.711682] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4357.775786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4357.982899] Lustre: 140952:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4358.076559] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4358.081793] Lustre: Skipped 1 previous similar message [ 4358.165506] Lustre: lustre-MDT0001: new disk, initializing [ 4358.266102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4358.307325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4358.315802] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4362.621718] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4369.974001] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4375.257554] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4382.477354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4382.795093] Lustre: lustre-OST0000: new disk, initializing [ 4382.798829] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4382.804467] Lustre: 142694:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4382.865661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4388.302427] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4392.003043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4392.013394] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4392.138379] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4395.570472] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4400.682852] Lustre: Failing over lustre-OST0000 [ 4400.819170] Lustre: server umount lustre-OST0000 complete [ 4418.015160] Lustre: 139175:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780454632/real 1780454632] req@0000000090b58b0b x1866941895573248/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1780454648 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4418.015160] Lustre: 139177:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780454632/real 1780454632] req@0000000004adff36 x1866941895574016/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1780454648 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4418.015261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4418.075134] Lustre: Skipped 1 previous similar message [ 4419.239190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4419.624672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4419.650096] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4419.914892] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4423.135533] Lustre: 139178:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780454637/real 1780454637] req@0000000052d617f4 x1866941895575040/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1780454653 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4423.182715] Lustre: 139178:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4426.243325] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4428.256613] Lustre: 139177:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780454642/real 1780454642] req@00000000e0ffc913 x1866941895576192/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1780454658 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4428.300551] Lustre: 139177:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4428.815882] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4428.816089] Lustre: lustre-OST0000: Recovery over after 0:09, of 3 clients 3 recovered and 0 were evicted. [ 4430.605087] Lustre: Failing over lustre-MDT0000 [ 4430.961226] Lustre: server umount lustre-MDT0000 complete [ 4433.920210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4433.941050] Lustre: Skipped 2 previous similar messages [ 4439.009195] LustreError: 141855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4439.052971] LustreError: 141855:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4443.375809] LustreError: 140960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4448.504032] LustreError: 140960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4448.534237] LustreError: 140960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4449.248534] Lustre: 139175:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780454663/real 1780454663] req@0000000004adff36 x1866941895586176/t0(0) o400->MGC192.168.203.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780454679 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4449.283212] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4449.508168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4451.615540] LustreError: 140958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4451.630511] LustreError: 140958:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4458.741543] LustreError: 142239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4458.771930] LustreError: 142239:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4459.784429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4459.856677] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4461.307307] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4463.811213] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4465.133317] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4465.140928] Lustre: Skipped 1 previous similar message [ 4465.166777] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4465.203165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4474.858332] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4474.865763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4474.884734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4477.185239] Lustre: server umount lustre-OST0000 complete [ 4480.480443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4480.483197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4480.490391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4480.516348] Lustre: Skipped 2 previous similar messages [ 4480.530332] Lustre: Skipped 3 previous similar messages [ 4485.612095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4485.615641] Lustre: Skipped 1 previous similar message [ 4486.604486] Lustre: server umount lustre-MDT0000 complete [ 4489.989965] LustreError: 140944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780454719 with bad export cookie 14770315811364379304 [ 4489.991533] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4490.005109] LustreError: 140944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4490.427632] Lustre: server umount lustre-MDT0001 complete [ 4499.134579] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 4501.491692] Key type lgssc unregistered [ 4501.767042] LNet: 146186:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4501.774816] LNetError: 146186:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4501.788479] LNet: Removed LNI 192.168.203.154@tcp [ 4502.537172] Key type .llcrypt unregistered [ 4502.546350] Key type ._llcrypt unregistered [ 4537.994773] Key type ._llcrypt registered [ 4537.999857] Key type .llcrypt registered [ 4538.152775] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4539.570618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4539.616278] alg: No test for adler32 (adler32-zlib) [ 4540.811582] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 4541.168401] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 4542.871288] Key type lgssc registered [ 4544.623232] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4554.199831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4554.226685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4555.808668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4560.671310] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4565.984046] LustreError: 147693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4566.005907] LustreError: 147693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4568.011544] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4571.111127] LustreError: 147694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4571.441153] LustreError: 147722:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4571.449797] LustreError: 147722:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4571.456565] LustreError: 147722:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 4571.716951] Lustre: server umount lustre-MDT0000 complete [ 4585.701947] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 4594.637953] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4628.704287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4629.049273] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4629.110049] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4629.180995] Lustre: lustre-MDT0000: new disk, initializing [ 4629.260738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4629.287324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4633.819282] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4643.378502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4643.548088] Lustre: 150878:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4643.638727] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4643.643138] Lustre: Skipped 1 previous similar message [ 4643.759953] Lustre: lustre-MDT0001: new disk, initializing [ 4643.885845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4643.963321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4643.981364] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4648.048549] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4655.518111] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4659.138684] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4666.252153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4666.509971] Lustre: lustre-OST0000: new disk, initializing [ 4666.513914] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4666.519976] Lustre: 152622:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4666.606554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4672.411441] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4675.643086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4675.662152] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4675.769905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4679.929558] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4685.045624] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4687.359259] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 22:48:35 (1780454915) [ 4945.890750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4945.904693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4945.919594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4947.423594] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4947.424732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4947.447144] Lustre: Skipped 2 previous similar messages [ 4949.970923] Lustre: server umount lustre-MDT0000 complete [ 4952.547319] LustreError: 150887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4952.558769] LustreError: 150887:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4953.135042] LustreError: 150870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780455183 with bad export cookie 13354643379670465123 [ 4953.138824] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4953.146601] LustreError: 150870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4957.663719] LustreError: 150885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4957.665225] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4957.705988] Lustre: Skipped 1 previous similar message [ 4957.711527] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4959.642826] Lustre: server umount lustre-MDT0001 complete [ 4969.812388] Lustre: server umount lustre-OST0000 complete [ 4978.084425] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 4986.228276] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5024.512198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5024.815735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5024.866877] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5024.982530] Lustre: lustre-MDT0000: new disk, initializing [ 5025.121341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5025.154515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5029.492775] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5040.461385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5040.568602] Lustre: 157140:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5040.630116] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5040.641121] Lustre: Skipped 1 previous similar message [ 5040.738840] Lustre: lustre-MDT0001: new disk, initializing [ 5040.890526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5040.933765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5040.943607] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5045.078121] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5052.484848] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5056.012173] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5063.512580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5063.806432] Lustre: lustre-OST0000: new disk, initializing [ 5063.815686] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5063.825531] Lustre: 158881:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5063.932309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5070.137108] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5073.457428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5073.476350] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5073.630576] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5078.063642] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5082.306184] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5082.562588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5086.570346] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5086.739184] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5089.258133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5089.263781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5089.287503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5089.298073] Lustre: Skipped 2 previous similar messages [ 5094.376251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5094.386410] Lustre: Skipped 1 previous similar message [ 5094.796556] Lustre: server umount lustre-OST0000 complete [ 5099.492812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5099.512357] Lustre: Skipped 1 previous similar message [ 5104.617433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5104.629845] Lustre: Skipped 2 previous similar messages [ 5104.639710] LustreError: 157147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5104.662887] LustreError: 157147:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5104.811575] Lustre: server umount lustre-MDT0000 complete [ 5108.568393] LustreError: 157133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780455338 with bad export cookie 13354643379670623953 [ 5108.572633] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5108.580414] LustreError: 157133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5109.066630] Lustre: server umount lustre-MDT0001 complete [ 5117.622345] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:55:46 (1780455346) [ 5134.821812] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 5144.520463] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5178.261373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5178.471628] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5178.513498] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5178.625821] Lustre: lustre-MDT0000: new disk, initializing [ 5178.743100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5178.765531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5183.335344] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5193.322743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5193.445367] Lustre: 163454:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5193.710908] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5197.552109] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5205.770266] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5209.480335] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5216.307889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5216.546309] Lustre: lustre-OST0000: new disk, initializing [ 5216.549098] Lustre: Skipped 1 previous similar message [ 5216.556104] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5216.560168] Lustre: Skipped 2 previous similar messages [ 5216.564918] Lustre: 165201:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5223.507280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5223.516439] Lustre: Skipped 1 previous similar message [ 5223.522251] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5223.705522] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5223.725827] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5232.143314] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5244.394808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5244.397412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5244.407383] LustreError: Skipped 1 previous similar message [ 5244.411019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5244.451556] Lustre: Skipped 2 previous similar messages [ 5246.914048] Lustre: server umount lustre-OST0000 complete [ 5265.375388] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5265.675316] Lustre: server umount lustre-MDT0000 complete [ 5269.575460] LustreError: 163446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780455499 with bad export cookie 13354643379670624835 [ 5269.577097] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5269.585620] LustreError: 163446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5270.143238] Lustre: server umount lustre-MDT0001 complete [ 5279.094333] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:58:27 (1780455507) [ 5295.585761] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 5304.461550] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5347.110691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5347.372698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5347.399487] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5347.457373] Lustre: lustre-MDT0000: new disk, initializing [ 5347.517369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5347.522029] Lustre: Skipped 2 previous similar messages [ 5347.534908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5351.735248] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5362.871346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5363.048529] Lustre: 169380:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5363.303984] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5367.657241] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5376.075703] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5380.652627] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5387.894623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5388.499221] Lustre: 171129:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5394.068996] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5396.287068] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5403.455573] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5414.882684] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5414.887989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5414.901701] LustreError: Skipped 1 previous similar message [ 5414.907620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5414.907627] Lustre: Skipped 7 previous similar messages [ 5414.951280] Lustre: Skipped 3 previous similar messages [ 5425.119664] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5425.277757] Lustre: server umount lustre-OST0000 complete [ 5429.728384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5435.360878] LustreError: 169392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5435.381149] LustreError: 169392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5435.435328] Lustre: server umount lustre-MDT0000 complete [ 5439.487027] LustreError: 169373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780455669 with bad export cookie 13354643379670625913 [ 5439.496909] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5439.500321] LustreError: 169373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5440.053447] Lustre: server umount lustre-MDT0001 complete [ 5452.020360] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 5454.979986] Key type lgssc unregistered [ 5455.318649] LNet: 172870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5455.324343] LNetError: 172870:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5455.350295] LNet: Removed LNI 192.168.203.154@tcp [ 5456.237162] Key type .llcrypt unregistered [ 5456.241227] Key type ._llcrypt unregistered [ 5478.158748] Key type ._llcrypt registered [ 5478.162492] Key type .llcrypt registered [ 5478.339621] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5479.510369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5479.545793] alg: No test for adler32 (adler32-zlib) [ 5480.643712] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 5480.908405] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 5482.704680] Key type lgssc registered [ 5484.887537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5494.387338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5494.423538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5496.410915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5501.234644] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5506.540142] LustreError: 174250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5506.565507] LustreError: 174250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5509.493781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5509.958054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5514.810831] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5521.404958] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5525.072673] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5531.517035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5531.895458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5538.672278] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5541.106320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5544.342645] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5565.922963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5565.931510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5565.952520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5566.945198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5566.945746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5566.949261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5566.996618] Lustre: Skipped 1 previous similar message [ 5568.838525] Lustre: server umount lustre-OST0000 complete [ 5576.674039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5576.688118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5576.715354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5579.106547] Lustre: server umount lustre-MDT0000 complete [ 5582.307950] LustreError: 174249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5582.343909] LustreError: 174249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5583.404786] LustreError: 174229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780455813 with bad export cookie 5420143086748063369 [ 5583.415294] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5583.846952] Lustre: server umount lustre-MDT0001 complete [ 5594.755080] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 5598.700814] Key type lgssc unregistered [ 5599.098963] LNet: 177450:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5599.110238] LNetError: 177450:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5600.167136] LNet: Removed LNI 192.168.203.154@tcp [ 5601.314142] Key type .llcrypt unregistered [ 5601.317245] Key type ._llcrypt unregistered [ 5628.972088] Key type ._llcrypt registered [ 5628.978232] Key type .llcrypt registered [ 5629.111591] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5630.088990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5630.131305] alg: No test for adler32 (adler32-zlib) [ 5631.450066] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 5631.763502] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 5633.455206] Key type lgssc registered [ 5635.133482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5644.566190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5644.589278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5646.032681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5650.659497] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5655.522926] LustreError: 178859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5655.540323] LustreError: 178859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5660.642144] LustreError: 178858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5662.182599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5662.712737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5667.724317] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5673.477603] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5677.791395] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5685.652504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5686.188962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5690.288399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5693.587218] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5699.845631] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5707.527713] Lustre: lustre-OST0000: Not available for connect from 192.168.203.54@tcp (stopping) [ 5707.744581] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5707.755871] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5710.825768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5710.837535] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5710.838540] Lustre: Skipped 1 previous similar message [ 5710.860954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5712.501320] Lustre: server umount lustre-OST0000 complete [ 5717.861423] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9a24c6fda000.ost_server_uuid 50 [ 5719.442754] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5734.369398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5734.376987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5734.391050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5734.401158] Lustre: Skipped 1 previous similar message [ 5740.514496] LustreError: 178858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5740.536598] LustreError: 178858:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5741.707245] Lustre: server umount lustre-MDT0000 complete [ 5745.633955] LustreError: 178859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5745.669756] LustreError: 178859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5745.861425] LustreError: 181468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780455975 with bad export cookie 17154953711158333228 [ 5745.866506] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5745.884321] LustreError: 181468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5746.540722] Lustre: server umount lustre-MDT0001 complete [ 5757.246529] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 23:06:25 (1780455985) [ 5765.169911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5765.655242] LustreError: 182342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5765.739324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5770.389564] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5775.332765] LustreError: 182343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5775.359490] LustreError: 182343:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5779.339267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5779.758355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5784.702582] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5791.049445] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5795.381327] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5802.081504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5802.438060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5808.417853] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5810.673167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5814.246362] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5821.432403] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5821.689752] Lustre: lustre-OST0001: new disk, initializing [ 5821.693554] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5821.701785] Lustre: 184843:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5821.806690] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5828.211232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5828.218033] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5828.233553] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5828.314664] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5836.006537] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5838.823890] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5838.827157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5838.835612] Lustre: Skipped 2 previous similar messages [ 5838.847911] Lustre: Skipped 3 previous similar messages [ 5843.952778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5843.961715] Lustre: Skipped 2 previous similar messages [ 5844.832914] Lustre: server umount lustre-OST0000 complete [ 5846.763981] LustreError: 183936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5851.505975] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5851.750412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5856.068232] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5856.360537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5874.660365] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5874.681708] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5874.683401] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5874.698204] Lustre: Skipped 1 previous similar message [ 5875.968886] Lustre: server umount lustre-OST0001 complete [ 5885.857217] Lustre: server umount lustre-MDT0000 complete [ 5889.353429] LustreError: 182323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780456119 with bad export cookie 17154953711158334292 [ 5889.361280] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5889.367486] LustreError: 182323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5889.707506] Lustre: server umount lustre-MDT0001 complete [ 5918.512910] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 23:09:07 (1780456147) [ 5925.481745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5925.964419] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5926.016830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5926.335971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5931.831108] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5940.819487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5941.000094] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5941.066626] Lustre: 188133:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5946.173877] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5952.315203] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5956.111481] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5963.267593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5963.401916] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5970.168674] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5972.192873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5976.324938] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5983.232501] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5983.364119] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5991.448115] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5993.280601] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5998.969811] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6003.171491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6003.182123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6003.215989] Lustre: Skipped 3 previous similar messages [ 6003.250572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6003.261091] Lustre: Skipped 5 previous similar messages [ 6008.637776] Lustre: server umount lustre-OST0000 complete [ 6008.805982] LustreError: 189761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6008.847137] LustreError: 189761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 6029.287594] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6029.289171] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6029.296858] LustreError: Skipped 1 previous similar message [ 6029.311077] Lustre: Skipped 1 previous similar message [ 6031.247981] Lustre: server umount lustre-OST0001 complete [ 6039.009463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6039.017099] Lustre: Skipped 3 previous similar messages [ 6041.632739] Lustre: server umount lustre-MDT0000 complete [ 6047.042715] LustreError: 190683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780456277 with bad export cookie 17154953711158335699 [ 6047.043746] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6047.056141] LustreError: 190683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6047.513838] Lustre: server umount lustre-MDT0001 complete [ 6074.946916] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 23:11:43 (1780456303) [ 6091.780436] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 6102.113532] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 6142.717979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6143.026669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6143.060208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6143.147554] Lustre: lustre-MDT0000: new disk, initializing [ 6143.230704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6143.240369] Lustre: Skipped 3 previous similar messages [ 6143.256609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6148.415912] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6160.456523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6160.597966] Lustre: 195750:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6160.642347] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6160.649558] Lustre: Skipped 1 previous similar message [ 6160.736931] Lustre: lustre-MDT0001: new disk, initializing [ 6160.893257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6160.905890] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6166.174976] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6175.936183] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6180.628912] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6190.694340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6191.146105] Lustre: lustre-OST0000: new disk, initializing [ 6191.154704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6191.171469] Lustre: 197490:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6198.663551] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6200.906181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6200.927825] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6201.098492] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6207.111690] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6212.502735] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6212.787635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6218.106348] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6218.336897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6221.792625] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6221.797122] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6221.811806] LustreError: Skipped 3 previous similar messages [ 6221.817180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6221.817190] Lustre: Skipped 1 previous similar message [ 6221.849526] Lustre: Skipped 4 previous similar messages [ 6226.541272] Lustre: server umount lustre-OST0000 complete [ 6236.445236] Lustre: server umount lustre-MDT0000 complete [ 6237.160349] LustreError: 196747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6237.196545] LustreError: 196747:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 6240.223671] LustreError: 198888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780456470 with bad export cookie 17154953711158337491 [ 6240.231164] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6240.238794] LustreError: 198888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6240.683879] Lustre: server umount lustre-MDT0001 complete [ 6248.535392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6253.375751] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6261.867376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6266.894267] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6273.514447] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6277.577375] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6287.155491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6287.906460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6287.920399] Lustre: Skipped 4 previous similar messages [ 6295.927563] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6302.580051] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6308.428510] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6308.673561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6313.854382] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6314.110422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6318.573467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6318.580861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6318.591547] Lustre: Skipped 2 previous similar messages [ 6323.087290] Lustre: server umount lustre-OST0000 complete [ 6329.622574] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6334.042913] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6363.946919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6369.573469] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6369.596472] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6371.750638] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6379.148474] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6382.124785] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6385.003985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.020486] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6390.255444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6390.257454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6390.272072] LustreError: Skipped 2 previous similar messages [ 6390.285669] Lustre: Skipped 7 previous similar messages [ 6393.331984] Lustre: server umount lustre-OST0000 complete [ 6404.019500] Lustre: server umount lustre-MDT0000 complete [ 6405.600249] LustreError: 199564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6405.609374] LustreError: 199564:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6407.915937] LustreError: 199544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780456637 with bad export cookie 17154953711158338373 [ 6407.925580] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6407.934565] LustreError: 199544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6408.389368] Lustre: server umount lustre-MDT0001 complete [ 6419.106119] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 23:17:27 (1780456647) [ 6426.737193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6432.567730] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6444.029665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6450.276315] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6457.768334] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6462.861259] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6471.615771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6479.460172] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6485.398514] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6490.559801] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6490.804625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6495.794826] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6496.048697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6502.902699] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6503.308650] Lustre: lustre-OST0001: new disk, initializing [ 6503.319476] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6503.333752] Lustre: 208012:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6509.249665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6509.272296] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6509.410963] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6511.432651] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6520.986891] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6527.525171] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6527.872968] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6533.127572] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6533.466393] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6540.261716] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6540.276636] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6540.297069] Lustre: Skipped 5 previous similar messages [ 6540.330805] LustreError: Skipped 1 previous similar message [ 6541.779399] Lustre: server umount lustre-OST0001 complete [ 6549.167255] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6549.468612] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6554.499517] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6554.713700] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6585.793654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6586.031323] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6586.038877] Lustre: Skipped 5 previous similar messages [ 6591.460158] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6591.475983] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6591.495056] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6591.498025] Lustre: Skipped 1 previous similar message [ 6592.443318] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6599.161567] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6601.233233] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6609.733781] Lustre: server umount lustre-OST0001 complete [ 6633.672613] Lustre: server umount lustre-OST0000 complete [ 6644.028847] Lustre: server umount lustre-MDT0000 complete [ 6648.610197] LustreError: 204994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780456878 with bad export cookie 17154953711158339332 [ 6648.614375] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6648.622449] LustreError: 204994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6654.939230] Lustre: server umount lustre-MDT0001 complete [ 6683.097738] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 23:21:51 (1780456911) [ 6690.218887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6690.639167] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6690.697646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6695.553885] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6705.425193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6705.677508] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6705.752813] Lustre: 212972:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6713.144616] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6721.701549] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6726.652496] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6735.677061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6735.869339] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6742.579604] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6750.134427] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6762.331097] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6762.554064] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6770.681631] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6776.788161] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6779.028865] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6783.316778] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6783.507688] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6792.447784] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6792.826846] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6797.159745] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a24c4473000.ost_server_uuid 50 [ 6798.931632] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9a24c4473000.ost_server_uuid in IDLE state after 0 sec [ 6800.137733] Lustre: Permanently deactivating lustre-OST0001 [ 6800.160187] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6804.855663] Lustre: Permanently reactivating lustre-OST0001 [ 6804.881171] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6804.888525] Lustre: Skipped 2 previous similar messages [ 6811.137036] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6811.142571] LustreError: Skipped 4 previous similar messages [ 6811.147527] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6811.158846] Lustre: Skipped 7 previous similar messages [ 6811.163589] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6811.171211] Lustre: Skipped 13 previous similar messages [ 6815.762295] Lustre: server umount lustre-OST0001 complete [ 6819.307662] LustreError: 214588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6819.331968] LustreError: 214588:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 39 previous similar messages [ 6826.640372] Lustre: server umount lustre-OST0000 complete [ 6837.388226] Lustre: server umount lustre-MDT0000 complete [ 6841.506568] LustreError: 215503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780457071 with bad export cookie 17154953711158340627 [ 6841.517171] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6841.523571] LustreError: 215503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6841.801453] Lustre: server umount lustre-MDT0001 complete [ 6867.817263] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 23:24:56 (1780457096) [ 6875.634910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6876.000957] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6876.091424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6876.099907] Lustre: Skipped 2 previous similar messages [ 6881.210925] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6891.365988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6891.422214] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6891.440784] Lustre: 219081:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6895.677981] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6903.702351] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6908.672323] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6918.385333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6918.669196] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6918.733360] Lustre: Permanently deactivating lustre-OST0000 [ 6918.757412] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6926.284645] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6927.485551] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6934.712737] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6943.816830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6943.947952] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6951.769890] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6952.047801] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6959.607568] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6961.907033] Lustre: Permanently reactivating lustre-OST0000 [ 6961.940242] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6961.952446] Lustre: Skipped 2 previous similar messages [ 6969.978562] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6969.979920] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 42 seconds. I think it's dead, and I am evicting it. exp 00000000e02dfc2f, cur 1780457199 deadline 1780457187 last 1780457157 [ 6969.993883] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6970.002508] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6970.072021] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6984.752022] Lustre: server umount lustre-OST0001 complete [ 7003.103361] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7003.502759] Lustre: server umount lustre-OST0000 complete [ 7014.867293] Lustre: server umount lustre-MDT0000 complete [ 7019.809158] LustreError: 219073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780457249 with bad export cookie 17154953711158342300 [ 7019.817339] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7019.839070] LustreError: 219073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7020.319750] Lustre: server umount lustre-MDT0001 complete [ 7032.916571] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 7036.431889] Key type lgssc unregistered [ 7036.692843] LNet: 223498:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7036.699429] LNetError: 223498:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7036.721081] LNet: Removed LNI 192.168.203.154@tcp [ 7037.696136] Key type .llcrypt unregistered [ 7037.698511] Key type ._llcrypt unregistered [ 7062.514268] Key type ._llcrypt registered [ 7062.516889] Key type .llcrypt registered [ 7062.612573] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 7063.903160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7064.120312] alg: No test for adler32 (adler32-zlib) [ 7065.312355] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 7065.549623] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 7067.201369] Key type lgssc registered [ 7068.639407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7080.962832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7081.002138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7082.684444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7088.644931] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7092.203320] LustreError: 224951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7092.227226] LustreError: 224951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7097.313863] LustreError: 224952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7098.942355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7099.127490] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7099.182291] Lustre: 224940:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7099.202631] Lustre: Permanently deactivating lustre-MDT0001 [ 7099.232612] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7099.688669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7104.849618] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7106.593133] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7111.180990] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7115.505072] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7123.137842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7123.546291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7124.582450] LustreError: 226565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7124.604692] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7127.662312] LustreError: 226565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7127.681666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7127.685299] LustreError: 226565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7130.084559] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7133.164190] LustreError: 226565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7133.183623] LustreError: 226565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7136.357165] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7143.402707] LustreError: 226565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7143.432785] LustreError: 226565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7144.265929] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7144.457340] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7149.558115] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7149.558686] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7151.515313] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7157.634925] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7160.525714] Lustre: Permanently reactivating lustre-MDT0001 [ 7160.551674] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7160.560430] Lustre: Skipped 2 previous similar messages [ 7167.264853] Lustre: lustre-MDT0001: Client cab118e6-44a3-4896-866a-9d975fb8fdf3 (at 192.168.203.54@tcp) reconnecting [ 7170.130055] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7170.145252] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7170.153470] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7170.173923] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7171.569848] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7173.051889] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7179.483461] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7179.778734] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7184.398378] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7184.643162] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7187.321547] Lustre: Permanently deactivating lustre-MDT0001 [ 7187.347765] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7187.354954] Lustre: Skipped 2 previous similar messages [ 7193.749232] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7203.006211] Lustre: server umount lustre-MDT0000 complete [ 7206.883556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7206.888510] LustreError: 228002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7206.916780] Lustre: Skipped 3 previous similar messages [ 7206.962306] LustreError: 228002:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7207.321125] LustreError: 224932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780457437 with bad export cookie 3325777251951810773 [ 7207.330045] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7207.339216] LustreError: 224932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7208.071522] Lustre: server umount lustre-MDT0001 complete [ 7212.891233] Lustre: server umount lustre-OST0000 complete [ 7226.073861] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:30:54 (1780457454) [ 7239.413393] Lustre: server umount lustre-OST0001 complete [ 7248.493156] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 7260.590162] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 7299.125687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7299.416883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7299.427465] Lustre: Skipped 2 previous similar messages [ 7299.455891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7299.583537] Lustre: lustre-MDT0000: new disk, initializing [ 7299.680703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7299.717886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7304.491986] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7316.085827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7316.288050] Lustre: 232592:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7316.336894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7316.341675] Lustre: Skipped 1 previous similar message [ 7316.480993] Lustre: lustre-MDT0001: new disk, initializing [ 7316.624746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7316.677758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7316.687591] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7321.071467] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7329.096821] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7333.176731] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7340.843396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7341.122172] Lustre: lustre-OST0000: new disk, initializing [ 7341.125465] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7341.134673] Lustre: 234337:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7341.274719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7348.023704] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7350.866997] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7350.875677] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7350.953247] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7355.783312] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7358.053316] LustreError: 232599:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7366.772628] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7366.961382] Lustre: lustre-OST0001: new disk, initializing [ 7366.967952] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7366.975622] Lustre: 235335:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7368.063080] LustreError: 232599:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7368.110436] LustreError: 235343:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7374.698812] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7377.030549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7377.038134] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7377.197634] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7378.175118] LustreError: 235343:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7382.710917] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7384.677467] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7387.639355] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7387.646877] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7387.654455] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7387.656887] LustreError: Skipped 1 previous similar message [ 7387.714877] Lustre: Skipped 1 previous similar message [ 7392.747762] LustreError: 234318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7392.762098] LustreError: 234318:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7392.849803] Lustre: server umount lustre-OST0001 complete [ 7412.706695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7412.708783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7412.730025] LustreError: Skipped 1 previous similar message [ 7412.739703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7412.781524] Lustre: Skipped 1 previous similar message [ 7412.846455] Lustre: Skipped 2 previous similar messages [ 7418.301949] Lustre: server umount lustre-OST0000 complete [ 7428.577020] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7428.617884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7428.627646] Lustre: Skipped 3 previous similar messages [ 7433.697068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7433.713431] Lustre: Skipped 1 previous similar message [ 7438.818567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7438.830647] Lustre: Skipped 1 previous similar message [ 7439.327212] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7439.737244] Lustre: server umount lustre-MDT0000 complete [ 7444.373620] LustreError: 232586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780457674 with bad export cookie 3325777251951813587 [ 7444.374593] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7444.417378] LustreError: 232586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7450.824455] Lustre: server umount lustre-MDT0001 complete [ 7462.204719] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 7464.760186] Key type lgssc unregistered [ 7465.056545] LNet: 237240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7465.065811] LNetError: 237240:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7465.089830] LNet: Removed LNI 192.168.203.154@tcp [ 7465.933724] Key type .llcrypt unregistered [ 7465.940618] Key type ._llcrypt unregistered [ 7505.396518] Key type ._llcrypt registered [ 7505.398462] Key type .llcrypt registered [ 7505.529757] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 7507.283985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7507.372955] alg: No test for adler32 (adler32-zlib) [ 7508.521626] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 7508.802563] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 7510.511151] Key type lgssc registered [ 7511.652839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7520.492282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7520.529920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7521.967915] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7522.015693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7522.209304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7527.175626] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7536.732487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7536.863464] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7536.920697] Lustre: 239432:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7537.398174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7542.905783] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7549.521931] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7554.797550] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7563.227321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7563.496852] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7563.791585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7571.111838] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7571.542104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7576.916701] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7632.864964] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7632.877318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7632.881446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7632.895671] Lustre: Skipped 1 previous similar message [ 7632.919632] Lustre: Skipped 1 previous similar message [ 7634.857069] Lustre: server umount lustre-OST0000 complete [ 7639.490807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7654.387758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7654.824188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7661.759732] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7669.627225] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7672.228815] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7672.253578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7672.254819] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7672.301700] Lustre: Skipped 1 previous similar message [ 7713.251750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7713.259165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7713.268580] LustreError: Skipped 2 previous similar messages [ 7713.293377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7713.308540] Lustre: Skipped 1 previous similar message [ 7715.972233] Lustre: server umount lustre-OST0000 complete [ 7721.954172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7721.979888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7721.991326] Lustre: Skipped 1 previous similar message [ 7722.000240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7726.622210] Lustre: server umount lustre-MDT0000 complete [ 7730.868085] LustreError: 239425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780457960 with bad export cookie 6493498505535137006 [ 7730.875083] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7730.886030] LustreError: 239425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7731.452637] Lustre: server umount lustre-MDT0001 complete [ 7744.623285] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 7747.877325] Key type lgssc unregistered [ 7748.261574] LNet: 244392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7748.269347] LNetError: 244392:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7748.293332] LNet: Removed LNI 192.168.203.154@tcp [ 7749.217454] Key type .llcrypt unregistered [ 7749.223540] Key type ._llcrypt unregistered [ 7773.423588] Key type ._llcrypt registered [ 7773.427229] Key type .llcrypt registered [ 7773.532346] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 7775.098234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7775.137196] alg: No test for adler32 (adler32-zlib) [ 7776.656489] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 7776.919763] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 7778.719337] Key type lgssc registered [ 7780.545499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7790.460807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7790.477895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7792.256751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7797.329617] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7802.857988] LustreError: 245792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7802.868335] LustreError: 245792:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7806.859663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7807.437936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7811.877874] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7817.979223] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7821.770610] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7828.871590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7829.200412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7835.574142] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7837.389411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7841.826682] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7868.402948] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7868.405410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7868.437747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7869.676309] Lustre: server umount lustre-OST0000 complete [ 7873.504117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7873.507943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7873.520746] Lustre: Skipped 2 previous similar messages [ 7873.533819] Lustre: Skipped 2 previous similar messages [ 7878.625041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7878.640494] Lustre: Skipped 1 previous similar message [ 7879.404483] Lustre: server umount lustre-MDT0000 complete [ 7882.821522] LustreError: 245772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780458112 with bad export cookie 10160669121272431787 [ 7882.845962] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7883.122418] Lustre: server umount lustre-MDT0001 complete [ 7892.274059] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 7895.528572] Key type lgssc unregistered [ 7895.939630] LNet: 249740:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7895.947697] LNetError: 249740:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7895.982121] LNet: Removed LNI 192.168.203.154@tcp [ 7896.550125] Key type .llcrypt unregistered [ 7896.553575] Key type ._llcrypt unregistered [ 7915.399312] Key type ._llcrypt registered [ 7915.401541] Key type .llcrypt registered [ 7915.558470] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 7917.136619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7917.172165] alg: No test for adler32 (adler32-zlib) [ 7918.285320] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 7918.566418] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 7920.287202] Key type lgssc registered [ 7921.648207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7930.187661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7930.249763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7931.894045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7936.355963] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7941.097176] LustreError: 250945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7941.113690] LustreError: 250945:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7944.875705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7945.301387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7949.393744] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7954.792692] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7958.632291] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7965.223893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7965.524188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7972.154328] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7976.879231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7977.994684] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7991.268900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7991.282634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7991.294970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7992.288986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7992.299486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7992.319508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7995.492039] Lustre: server umount lustre-OST0000 complete [ 8001.504320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8001.519356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8001.547160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8001.553862] Lustre: Skipped 1 previous similar message [ 8006.482751] Lustre: server umount lustre-MDT0000 complete [ 8007.649024] LustreError: 250945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8007.656160] LustreError: 250945:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8010.079256] LustreError: 250925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780458240 with bad export cookie 6994515543293543280 [ 8010.095189] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8010.412193] Lustre: server umount lustre-MDT0001 complete [ 8020.396618] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 8023.241631] Key type lgssc unregistered [ 8023.519098] LNet: 254268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8023.527783] LNetError: 254268:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8023.551486] LNet: Removed LNI 192.168.203.154@tcp [ 8024.461751] Key type .llcrypt unregistered [ 8024.464051] Key type ._llcrypt unregistered [ 8048.933671] Key type ._llcrypt registered [ 8048.936211] Key type .llcrypt registered [ 8049.030716] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 8050.115405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8050.254728] alg: No test for adler32 (adler32-zlib) [ 8051.412851] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 8051.669789] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 8053.343184] Key type lgssc registered [ 8054.567408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8062.160889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8062.189702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8063.682207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8067.975530] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8073.194205] LustreError: 255668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8073.229137] LustreError: 255668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8076.755487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8077.219893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8081.389495] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8087.221068] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8091.163979] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8097.178289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8097.460635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8102.695634] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8108.717769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8108.765281] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8133.604028] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8133.610966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8133.622853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8134.629796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8134.630560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8134.655707] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8134.659735] Lustre: Skipped 1 previous similar message [ 8138.376910] Lustre: server umount lustre-OST0000 complete [ 8143.840803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8143.847179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8143.863087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8148.063290] Lustre: server umount lustre-MDT0000 complete [ 8149.984520] LustreError: 255668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8150.006602] LustreError: 255668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8151.434493] LustreError: 255647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780458381 with bad export cookie 13630648807553925383 [ 8151.449352] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8151.451058] LustreError: 255647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8151.747480] Lustre: server umount lustre-MDT0001 complete [ 8162.409578] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 8165.022381] Key type lgssc unregistered [ 8165.300973] LNet: 259617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8165.310189] LNetError: 259617:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8165.337825] LNet: Removed LNI 192.168.203.154@tcp [ 8166.221995] Key type .llcrypt unregistered [ 8166.224268] Key type ._llcrypt unregistered [ 8181.424590] Key type ._llcrypt registered [ 8181.428019] Key type .llcrypt registered [ 8181.557774] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 8182.417458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8182.505092] alg: No test for adler32 (adler32-zlib) [ 8183.523270] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 8183.744265] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 8185.415145] Key type lgssc registered [ 8186.233746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8193.244946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8193.264185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8194.973421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8198.984628] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8204.257692] LustreError: 260828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8204.276376] LustreError: 260828:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8206.786745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8207.338843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8211.738208] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8217.162679] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8221.275054] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8228.344961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8228.725880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8235.027448] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8239.992623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8240.371709] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8253.414167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8253.425109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8253.452730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8255.457693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8255.466857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8255.477981] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8255.518515] Lustre: Skipped 1 previous similar message [ 8258.438017] Lustre: server umount lustre-OST0000 complete [ 8263.648703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8263.661714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8263.699891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8265.701827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8265.719540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8265.733288] Lustre: Skipped 1 previous similar message [ 8268.296315] Lustre: server umount lustre-MDT0000 complete [ 8270.821484] LustreError: 260827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8270.862495] LustreError: 260827:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8272.245383] LustreError: 260794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780458502 with bad export cookie 15595105698437135427 [ 8272.251433] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8272.259943] LustreError: 260794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8272.719433] Lustre: server umount lustre-MDT0001 complete [ 8283.387814] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 8285.989964] Key type lgssc unregistered [ 8286.271575] LNet: 264172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8286.279440] LNetError: 264172:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8286.295848] LNet: Removed LNI 192.168.203.154@tcp [ 8287.168276] Key type .llcrypt unregistered [ 8287.170868] Key type ._llcrypt unregistered [ 8298.163992] Key type ._llcrypt registered [ 8298.166818] Key type .llcrypt registered [ 8298.298058] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8315.130552] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 8329.194146] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 8330.258754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8330.337649] alg: No test for adler32 (adler32-zlib) [ 8331.372716] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 8331.544983] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 8333.191641] Key type lgssc registered [ 8334.323710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8362.838580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8362.875569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8364.100583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8364.131543] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8364.199888] Lustre: lustre-MDT0000: new disk, initializing [ 8364.285344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8364.346728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8368.298779] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8379.728410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8379.974642] Lustre: 267672:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8380.053193] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8380.058165] Lustre: Skipped 1 previous similar message [ 8380.144131] Lustre: lustre-MDT0001: new disk, initializing [ 8380.298292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8380.345063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8380.360563] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8385.672948] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8395.198869] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8399.532886] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8406.720869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8407.260431] Lustre: lustre-OST0000: new disk, initializing [ 8407.263516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8407.270790] Lustre: 269416:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8407.486658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8412.760712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8412.775377] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8412.983276] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8413.557448] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8420.548279] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8424.660766] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8424.840721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8428.926053] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8429.147340] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8433.636842] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8433.643706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8433.652775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8433.657431] Lustre: Skipped 1 previous similar message [ 8433.671227] Lustre: Skipped 1 previous similar message [ 8438.766551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8438.775823] Lustre: Skipped 1 previous similar message [ 8443.878295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8443.882553] Lustre: Skipped 1 previous similar message [ 8445.407226] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8445.663854] Lustre: server umount lustre-OST0000 complete [ 8452.065990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8452.072826] LustreError: Skipped 1 previous similar message [ 8452.079407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8452.092162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8454.119942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8455.828770] Lustre: server umount lustre-MDT0000 complete [ 8459.810748] LustreError: 269415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780458689 with bad export cookie 11649899232858134993 [ 8459.816843] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8459.830799] LustreError: 269415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8460.266080] Lustre: server umount lustre-MDT0001 complete [ 8469.906510] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 23:51:38 (1780458698) [ 8476.882616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8477.493677] LustreError: 271681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8477.525594] LustreError: 271681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8477.615144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8481.954219] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8482.788646] LustreError: 271682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8487.908125] LustreError: 271681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8490.020790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8490.399506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8494.810907] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8500.519965] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8504.673822] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8511.750586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8512.271715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8518.366872] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8524.551311] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8536.032494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8536.039294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8536.068930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8536.088505] Lustre: Skipped 2 previous similar messages [ 8538.086686] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8540.048096] Lustre: server umount lustre-OST0000 complete [ 8546.787204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8546.796299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8546.819771] Lustre: Skipped 1 previous similar message [ 8546.828183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8546.837536] Lustre: Skipped 2 previous similar messages [ 8549.813533] Lustre: server umount lustre-MDT0000 complete [ 8553.455321] LustreError: 273835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8553.482277] LustreError: 273835:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8553.523667] LustreError: 271662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780458783 with bad export cookie 11649899232858135875 [ 8553.538898] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8553.952713] Lustre: server umount lustre-MDT0001 complete [ 8565.053538] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 8567.664377] Key type lgssc unregistered [ 8567.906573] LNet: 274859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8567.917480] LNetError: 274859:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8568.933678] LNet: Removed LNI 192.168.203.154@tcp [ 8569.835041] Key type .llcrypt unregistered [ 8569.839105] Key type ._llcrypt unregistered [ 8595.196903] Key type ._llcrypt registered [ 8595.199449] Key type .llcrypt registered [ 8595.344223] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 8610.260223] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 8611.232545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8611.265769] alg: No test for adler32 (adler32-zlib) [ 8612.338862] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 8612.541782] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 8614.263162] Key type lgssc registered [ 8615.384734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8662.003444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8662.028089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8663.404782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8663.486675] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8663.595184] Lustre: lustre-MDT0000: new disk, initializing [ 8663.716184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8663.754573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8667.675570] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8678.212356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8678.315583] Lustre: 278652:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8678.389383] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8678.396673] Lustre: Skipped 1 previous similar message [ 8678.478367] Lustre: lustre-MDT0001: new disk, initializing [ 8678.562145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8678.598705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8678.614541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8683.166053] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8690.643573] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8694.494701] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8701.491049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8701.764307] Lustre: lustre-OST2710: new disk, initializing [ 8701.768637] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8701.775856] Lustre: 280394:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8701.874826] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8707.672479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8707.684312] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8707.842162] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8708.243166] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8717.070589] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8725.303308] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8725.498145] Lustre: lustre-OST03e8: new disk, initializing [ 8725.502453] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8725.516214] Lustre: 281344:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8725.609246] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8731.422396] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8732.763096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8732.774932] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8732.843918] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8737.189210] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8745.029550] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8745.238380] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8749.561506] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8749.775959] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8753.720207] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8753.956281] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8757.867975] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8758.106301] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8763.874796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8763.885102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8763.894299] Lustre: Skipped 2 previous similar messages [ 8765.533312] Lustre: server umount lustre-MDT0000 complete [ 8769.008807] LustreError: 278657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8769.034454] LustreError: 278657:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8769.398795] LustreError: 278643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780458999 with bad export cookie 18142113899150906516 [ 8769.400980] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8769.423511] LustreError: 278643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8769.930658] Lustre: server umount lustre-MDT0001 complete [ 8784.795315] Lustre: server umount lustre-OST2710 complete [ 8798.770414] Lustre: server umount lustre-OST03e8 complete [ 8816.924760] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 8826.923383] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 8863.013650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8863.390676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8863.428132] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8863.558535] Lustre: lustre-MDT0000: new disk, initializing [ 8863.667625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8863.687216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8869.966138] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8884.040520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8884.151618] Lustre: 286383:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8884.284905] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8884.287789] Lustre: Skipped 1 previous similar message [ 8884.360234] Lustre: lustre-MDT0001: new disk, initializing [ 8884.467217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8884.494321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8884.507736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8889.093205] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8897.537981] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8902.186231] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8910.176396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8910.472406] Lustre: lustre-OST0000: new disk, initializing [ 8910.476597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8910.487473] Lustre: 288125:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8910.617314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8916.527606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8916.537189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8916.662079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8916.878099] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8924.593264] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8929.467911] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8929.694469] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8934.082539] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8934.295825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8937.441054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8937.444193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8937.458505] Lustre: Skipped 3 previous similar messages [ 8937.471611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8937.486258] Lustre: Skipped 2 previous similar messages [ 8942.564195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8942.577556] Lustre: Skipped 1 previous similar message [ 8947.689590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8947.702148] Lustre: Skipped 1 previous similar message [ 8950.239121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8950.394178] Lustre: server umount lustre-OST0000 complete [ 8956.385536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8956.393320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8956.427147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8960.852656] Lustre: server umount lustre-MDT0000 complete [ 8963.046887] LustreError: 286394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8963.095676] LustreError: 286394:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8964.707338] LustreError: 286374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780459194 with bad export cookie 18142113899150908189 [ 8964.715443] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8964.727844] LustreError: 286374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8965.022094] Lustre: server umount lustre-MDT0001 complete [ 8974.389739] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 00:00:03 (1780459203) [ 8975.883772] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8977.787301] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 00:00:06 (1780459206) [ 8981.203273] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 9007.941134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9008.199951] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9008.221906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9008.386110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9012.880622] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9022.239592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9022.376066] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9022.413164] Lustre: 291820:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9022.486624] LustreError: 291830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9026.871993] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9048.633332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9048.910869] Lustre: Denying initial registration attempt from nid 192.168.203.154@tcp, specified as failover [ 9048.938918] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9048.951988] LustreError: 293254:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 9048.962107] LustreError: 293254:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 9048.969904] LustreError: 293254:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9049.071169] Lustre: server umount lustre-OST0000 complete [ 9049.073318] LustreError: 293254:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9058.784660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9058.786525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9058.799394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9058.809339] Lustre: Skipped 2 previous similar messages [ 9058.834656] Lustre: Skipped 3 previous similar messages [ 9061.640854] Lustre: server umount lustre-MDT0000 complete [ 9063.904394] LustreError: 292862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9063.922237] LustreError: 292862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9065.818507] LustreError: 291810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780459295 with bad export cookie 18142113899150909078 [ 9065.819682] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9065.838346] LustreError: 291810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9066.213239] Lustre: server umount lustre-MDT0001 complete [ 9078.947937] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 9082.143412] Key type lgssc unregistered [ 9082.592621] LNet: 294359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9082.614360] LNetError: 294359:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9082.646665] LNet: Removed LNI 192.168.203.154@tcp [ 9083.719187] Key type .llcrypt unregistered [ 9083.726829] Key type ._llcrypt unregistered [ 9096.628539] Key type ._llcrypt registered [ 9096.630333] Key type .llcrypt registered [ 9096.717400] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 9097.694293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9097.715075] alg: No test for adler32 (adler32-zlib) [ 9098.863977] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 9099.153267] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 9100.863680] Key type lgssc registered [ 9101.746447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9149.149593] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 9158.033385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9158.054338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9159.480933] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9159.538779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9159.849427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9165.194453] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9175.511463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9175.682448] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9175.719350] Lustre: 297091:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9175.960703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9180.417378] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9185.658667] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9189.350081] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9209.006216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9209.184448] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9209.506545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9215.993644] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9221.568148] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9225.697632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9225.713903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9225.730556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9226.211416] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9226.224565] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9226.254447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9226.264804] Lustre: Skipped 1 previous similar message [ 9230.238357] Lustre: server umount lustre-OST0000 complete [ 9236.455607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9236.461588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9236.468051] Lustre: Skipped 1 previous similar message [ 9236.477298] Lustre: Skipped 1 previous similar message [ 9239.967389] Lustre: server umount lustre-MDT0000 complete [ 9243.307497] LustreError: 297081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780459473 with bad export cookie 10308994009012742687 [ 9243.313587] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9243.318649] LustreError: 297081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9243.613919] Lustre: server umount lustre-MDT0001 complete [ 9255.106629] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 9258.056943] Key type lgssc unregistered [ 9258.349111] LNet: 300306:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9258.365619] LNetError: 300306:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9258.390401] LNet: Removed LNI 192.168.203.154@tcp [ 9259.158887] Key type .llcrypt unregistered [ 9259.164608] Key type ._llcrypt unregistered [ 9284.420072] Key type ._llcrypt registered [ 9284.421452] Key type .llcrypt registered [ 9284.513545] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 9285.608868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9285.636582] alg: No test for adler32 (adler32-zlib) [ 9286.886229] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 9287.156428] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 9288.823593] Key type lgssc registered [ 9290.516646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9298.502986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9298.536034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9300.054333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9304.433589] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9310.177703] LustreError: 301708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9310.201185] LustreError: 301708:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9312.197730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9312.543396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9316.607856] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9323.004700] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9326.721846] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9336.636111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9337.006153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9344.096479] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9350.553843] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9360.864848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9360.870314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9360.888829] Lustre: Skipped 1 previous similar message [ 9360.905165] Lustre: Skipped 2 previous similar messages [ 9363.722404] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.54@tcp (stopping) [ 9365.984523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9365.996960] Lustre: Skipped 2 previous similar messages [ 9366.229842] Lustre: server umount lustre-MDT0000 complete [ 9368.823875] LustreError: 302727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9370.223140] LustreError: 301689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780459600 with bad export cookie 2346385666126323527 [ 9370.231434] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9370.241724] LustreError: 301689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9371.109309] LustreError: 301702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9371.119432] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9371.145754] Lustre: Skipped 1 previous similar message [ 9371.153692] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9373.935808] LustreError: 301702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9376.634831] Lustre: server umount lustre-MDT0001 complete [ 9382.896470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9391.744752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9395.679859] LustreError: 301017:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000060975b88 x1866947118535808/t0(0) o250->MGC192.168.203.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9396.055934] LustreError: 303314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9396.076846] LustreError: 303314:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9396.191243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9396.275365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9400.620985] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9405.414954] LustreError: 303315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9405.428297] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9405.436082] LustreError: 303315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9408.826850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9409.289101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9413.580952] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9414.636042] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9418.678779] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9421.391254] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 9425.219820] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9427.039755] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9435.106229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9435.106660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9435.120346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9435.134402] Lustre: Skipped 3 previous similar messages [ 9438.924263] Lustre: server umount lustre-OST0000 complete [ 9445.346245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9445.350844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9445.351142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9445.371626] Lustre: Skipped 2 previous similar messages [ 9445.404571] Lustre: Skipped 1 previous similar message [ 9449.544229] Lustre: server umount lustre-MDT0000 complete [ 9450.466834] LustreError: 304958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9450.498932] LustreError: 304958:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9453.283995] LustreError: 301689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780459683 with bad export cookie 2346385666126331038 [ 9453.284802] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9453.294095] LustreError: 301689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9453.646641] Lustre: server umount lustre-MDT0001 complete [ 9465.699434] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 9468.784762] Key type lgssc unregistered [ 9469.120129] LNet: 307390:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9469.139249] LNetError: 307390:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9469.166836] LNet: Removed LNI 192.168.203.154@tcp [ 9469.977556] Key type .llcrypt unregistered [ 9469.979969] Key type ._llcrypt unregistered [ 9500.106374] Key type ._llcrypt registered [ 9500.111174] Key type .llcrypt registered [ 9500.233554] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 9501.149746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9501.237527] alg: No test for adler32 (adler32-zlib) [ 9502.304342] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 9502.551361] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 9504.231225] Key type lgssc registered [ 9505.686524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9514.126064] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9514.159053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9515.754691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9520.303195] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9525.223621] LustreError: 308789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9525.238605] LustreError: 308789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9530.006478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9530.435800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9534.302346] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9539.446889] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9543.613763] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9550.824641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9550.825393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9550.826571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9550.849391] Lustre: Skipped 1 previous similar message [ 9553.026876] Lustre: server umount lustre-MDT0000 complete [ 9555.936740] LustreError: 308784:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9555.960544] LustreError: 308784:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9556.964931] LustreError: 310134:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9557.020058] LustreError: 310134:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9557.765994] LustreError: 308769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780459787 with bad export cookie 474959553370844492 [ 9557.793804] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9557.799604] LustreError: 308769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9558.316584] Lustre: server umount lustre-MDT0001 complete [ 9576.270778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9576.661729] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9576.689443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9577.065739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9581.719509] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9591.243486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9591.304059] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9591.324516] Lustre: 311150:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9591.493696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9595.924692] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9602.342856] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9605.419191] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9615.499423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9615.701382] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9615.709155] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9615.716994] LustreError: 312810:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 9615.722222] LustreError: 312810:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 9615.726933] LustreError: 312810:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9615.792357] Lustre: server umount lustre-OST0000 complete [ 9615.799711] LustreError: 312810:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9625.714075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9625.895710] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9626.066592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9632.156605] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9634.911098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9637.563154] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9645.357173] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9645.472166] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9645.584430] Lustre: lustre-OST0001: new disk, initializing [ 9645.592567] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9645.602117] Lustre: 314236:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9645.766258] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9651.785320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9651.800181] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9651.893408] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9651.964654] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9656.955208] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9662.433259] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9662.438877] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9662.444666] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9662.453496] Lustre: Skipped 1 previous similar message [ 9662.462212] Lustre: Skipped 1 previous similar message [ 9665.555187] Lustre: server umount lustre-OST0001 complete [ 9667.559918] LustreError: 313277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9667.571548] LustreError: 313277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9670.624758] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9670.631339] LustreError: Skipped 1 previous similar message [ 9670.637628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9670.658897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9670.664208] Lustre: Skipped 1 previous similar message [ 9672.675536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9672.679274] LustreError: 313277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9672.688214] Lustre: Skipped 2 previous similar messages [ 9674.822316] Lustre: server umount lustre-OST0000 complete [ 9677.792401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9677.798154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9677.803258] Lustre: Skipped 2 previous similar messages [ 9688.033063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9688.039101] Lustre: Skipped 4 previous similar messages [ 9692.127557] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9692.288155] Lustre: server umount lustre-MDT0000 complete [ 9693.152649] LustreError: 312173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9693.174339] LustreError: 312173:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9695.483244] LustreError: 315330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780459925 with bad export cookie 474959553370845038 [ 9695.488166] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9695.494039] LustreError: 315330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9695.801627] Lustre: server umount lustre-MDT0001 complete [ 9706.815342] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 9709.385927] Key type lgssc unregistered [ 9709.636133] LNet: 316065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9709.643236] LNetError: 316065:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9709.684700] LNet: Removed LNI 192.168.203.154@tcp [ 9710.467090] Key type .llcrypt unregistered [ 9710.472559] Key type ._llcrypt unregistered [ 9782.307466] Key type ._llcrypt registered [ 9782.312503] Key type .llcrypt registered [ 9782.429286] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 9798.707343] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 9799.666143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9799.704325] alg: No test for adler32 (adler32-zlib) [ 9800.665676] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 9800.831358] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 9802.487541] Key type lgssc registered [ 9803.402735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9839.865701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9839.898764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9841.177386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9841.207937] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9841.279974] Lustre: lustre-MDT0000: new disk, initializing [ 9841.381635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9841.413101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9844.680938] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9853.768458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9853.893427] Lustre: 320965:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9853.949821] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9853.955199] Lustre: Skipped 1 previous similar message [ 9854.041137] Lustre: lustre-MDT0001: new disk, initializing [ 9854.152282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9854.208609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9854.226228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9858.019155] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9865.146740] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9869.208820] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9876.067190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9876.384523] Lustre: lustre-OST0000: new disk, initializing [ 9876.389444] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9876.396711] Lustre: 322707:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9876.488547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9881.649192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9881.654950] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9881.760755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9882.021207] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9888.226278] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9891.743666] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9891.884592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9895.095491] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9895.282572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9897.445226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9897.448230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9897.450325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9897.455590] LustreError: Skipped 1 previous similar message [ 9897.489354] Lustre: Skipped 1 previous similar message [ 9902.571585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9902.588253] Lustre: Skipped 2 previous similar messages [ 9902.922548] Lustre: server umount lustre-OST0000 complete [ 9907.681796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9907.684613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9907.690107] Lustre: Skipped 1 previous similar message [ 9912.800326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9912.804776] Lustre: Skipped 2 previous similar messages [ 9913.031984] Lustre: server umount lustre-MDT0000 complete [ 9916.445834] LustreError: 320957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780460146 with bad export cookie 18196206050740525346 [ 9916.451549] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9916.464782] LustreError: 320957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9916.830326] Lustre: server umount lustre-MDT0001 complete [ 9925.529631] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 00:15:54 (1780460154) [ 9933.605080] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 00:16:02 (1780460162) [ 9942.079725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9942.521829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9946.517963] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9947.622909] LustreError: 325316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9947.655041] LustreError: 325316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9952.737785] LustreError: 325315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9954.776224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9955.142358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9959.577995] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9964.968110] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9969.451544] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9975.733752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9976.188786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9982.394758] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9988.576322] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9996.720303] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9998.836575] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10000.541694] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10002.155426] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10004.076254] Lustre: Failing over lustre-MDT0000 [10004.406264] Lustre: server umount lustre-MDT0000 complete [10006.496131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10006.502976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10006.516877] LustreError: 325315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10011.880837] LustreError: 326474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10011.902419] LustreError: 326474:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10017.014522] LustreError: 325312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10017.038365] LustreError: 325312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10022.750848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10022.902526] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10023.193775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10023.241924] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10027.260068] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10027.349824] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10028.550220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10028.576734] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10028.652175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10035.769316] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10037.612915] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10042.294735] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10044.566908] Lustre: Failing over lustre-MDT0000 [10045.062415] Lustre: server umount lustre-MDT0000 complete [10048.271366] LustreError: 325311:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10048.298934] LustreError: 325311:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10048.997115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10049.002706] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10049.032875] Lustre: Skipped 4 previous similar messages [10064.289437] Lustre: 319191:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780460278/real 1780460278] req@0000000099782bb8 x1866947656502912/t0(0) o400->MGC192.168.203.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780460294 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10064.330359] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10064.341697] LustreError: 325315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10064.364320] LustreError: 325315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [10064.655628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10074.604911] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfc85e8feda33e3b7 [10074.635827] Lustre: MGC192.168.203.154@tcp: Connection restored to 0@lo (at 0@lo) [10074.645737] Lustre: Skipped 2 previous similar messages [10074.926632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10075.000510] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10079.191589] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10079.986661] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10080.258269] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10080.337668] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10080.401968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10087.850917] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10090.004740] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10100.711655] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10100.722387] Lustre: Skipped 1 previous similar message [10100.722419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10100.746113] Lustre: Skipped 2 previous similar messages [10104.056401] Lustre: server umount lustre-MDT0000 complete [10105.834809] LustreError: 326025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10105.877298] LustreError: 326025:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [10108.050351] LustreError: 325295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780460338 with bad export cookie 18196206050740528055 [10108.058429] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10108.067620] LustreError: 325295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10108.364422] Lustre: server umount lustre-MDT0001 complete [10123.598192] Lustre: server umount lustre-OST0000 complete [10138.987120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10139.535849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10143.587695] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10152.351126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10157.260468] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10163.440709] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10167.749235] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10175.319316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10181.413427] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10184.822805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10187.279970] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10193.376543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10193.382989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10193.429223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10199.405292] Lustre: server umount lustre-OST0000 complete [10204.128362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10204.133325] LustreError: Skipped 1 previous similar message [10209.536677] Lustre: server umount lustre-MDT0000 complete [10210.786723] LustreError: 331091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10210.816484] LustreError: 331091:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10213.098652] LustreError: 331072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780460443 with bad export cookie 18196206050740528804 [10213.106175] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10213.109311] LustreError: 331072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10213.423534] Lustre: server umount lustre-MDT0001 complete [10225.760993] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [10228.703840] Key type lgssc unregistered [10228.957829] LNet: 334265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10228.973195] LNetError: 334265:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10228.994164] LNet: Removed LNI 192.168.203.154@tcp [10230.016551] Key type .llcrypt unregistered [10230.021195] Key type ._llcrypt unregistered [10260.258735] Key type ._llcrypt registered [10260.261581] Key type .llcrypt registered [10260.371486] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [10261.794737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10261.823111] alg: No test for adler32 (adler32-zlib) [10262.950323] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [10263.175667] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [10264.895317] Key type lgssc registered [10266.385916] Lustre: Echo OBD driver; http://www.lustre.org/ [10274.810988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10274.831367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10276.512343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10281.115634] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10286.051814] LustreError: 335766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10286.083820] LustreError: 335766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10290.538091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10291.263765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10296.894367] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10303.265163] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10307.255431] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10315.462818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10316.012938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10321.204840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10323.568132] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10330.352403] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10341.860212] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10341.862264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10341.873698] Lustre: Skipped 1 previous similar message [10341.891871] Lustre: Skipped 2 previous similar messages [10346.942487] Lustre: server umount lustre-MDT0000 complete [10346.979845] LustreError: 336684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10347.006045] LustreError: 336684:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10350.355556] LustreError: 335746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780460580 with bad export cookie 5222139717828331458 [10350.369159] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10350.629755] Lustre: server umount lustre-MDT0001 complete [10364.407891] Lustre: server umount lustre-OST0000 complete [10377.927801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10378.322654] LustreError: 339139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10378.384300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10383.119860] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10383.841546] LustreError: 339140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10388.967232] LustreError: 339139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10392.170396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10392.674756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10398.188579] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10405.419891] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10410.720842] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10418.646554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10419.037341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10423.156559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10425.563466] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10431.647508] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10447.843391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10447.862506] Lustre: Skipped 1 previous similar message [10447.873125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10447.879310] Lustre: Skipped 2 previous similar messages [10450.431066] Lustre: server umount lustre-MDT0000 complete [10452.963378] LustreError: 340051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10452.997316] LustreError: 340051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10454.386606] LustreError: 340757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780460684 with bad export cookie 5222139717828332571 [10454.406575] LustreError: 340757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10454.411994] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10454.837445] Lustre: server umount lustre-MDT0001 complete [10468.387233] Lustre: server umount lustre-OST0000 complete [10484.393217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10484.972429] LustreError: 342710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10485.076937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10489.775672] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10498.640414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10503.206827] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10508.889601] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10512.599343] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10520.085270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10520.488192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10520.496363] Lustre: Skipped 1 previous similar message [10526.699639] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10529.729064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10532.425942] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10538.977634] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10538.995832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10539.015833] Lustre: Skipped 1 previous similar message [10539.025732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10543.474773] Lustre: server umount lustre-OST0000 complete [10550.241155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10550.246313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10550.254918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10550.254926] Lustre: Skipped 2 previous similar messages [10550.277713] Lustre: Skipped 2 previous similar messages [10553.307453] Lustre: server umount lustre-MDT0000 complete [10555.361884] LustreError: 343707:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10555.398402] LustreError: 343707:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10557.123479] LustreError: 342691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780460787 with bad export cookie 5222139717828333579 [10557.127148] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10557.137662] LustreError: 342691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10557.601363] Lustre: server umount lustre-MDT0001 complete [10571.024303] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [10574.535165] Key type lgssc unregistered [10574.979692] LNet: 345887:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10574.984820] LNetError: 345887:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10576.038837] LNet: Removed LNI 192.168.203.154@tcp [10577.474425] Key type .llcrypt unregistered [10577.476478] Key type ._llcrypt unregistered [10607.433699] Key type ._llcrypt registered [10607.437519] Key type .llcrypt registered [10607.547275] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [10608.487889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10608.612697] alg: No test for adler32 (adler32-zlib) [10609.874255] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [10610.156935] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [10611.847262] Key type lgssc registered [10613.446751] Lustre: Echo OBD driver; http://www.lustre.org/ [10621.596912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10621.619675] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10621.629991] LustreError: 347314:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10621.649222] LustreError: 347314:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10621.655247] LustreError: 347314:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10621.661240] LustreError: 347314:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10621.669884] LustreError: 347314:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10629.229229] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10629.240660] LustreError: 347712:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10629.255962] LustreError: 347712:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10629.261955] LustreError: 347712:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10629.269922] LustreError: 347712:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10629.279644] LustreError: 347712:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10648.701780] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [10651.345970] Key type lgssc unregistered [10651.599857] LNet: 348698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10651.608726] LNetError: 348698:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10651.655861] LNet: Removed LNI 192.168.203.154@tcp [10652.490823] Key type .llcrypt unregistered [10652.493357] Key type ._llcrypt unregistered [10669.197239] Key type ._llcrypt registered [10669.200358] Key type .llcrypt registered [10669.317127] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [10683.937646] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [10684.977020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10685.066765] alg: No test for adler32 (adler32-zlib) [10686.085497] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [10686.314577] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [10687.999287] Key type lgssc registered [10689.138736] Lustre: Echo OBD driver; http://www.lustre.org/ [10726.093987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10726.154696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10727.430696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10727.521731] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10727.646675] Lustre: lustre-MDT0000: new disk, initializing [10727.734523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10727.746636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10732.055918] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10742.851572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10743.038977] Lustre: 351819:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10743.138925] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10743.142435] Lustre: Skipped 1 previous similar message [10743.232844] Lustre: lustre-MDT0001: new disk, initializing [10743.316573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10743.356567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10743.372991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10747.681876] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10756.148527] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10760.841402] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10768.384299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10768.722111] Lustre: lustre-OST0000: new disk, initializing [10768.728865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10768.739370] Lustre: 353563:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10768.834828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10775.245815] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10777.692062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10777.704877] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10777.821277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10782.306894] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10786.793254] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10787.030230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10791.377053] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10791.692630] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10798.573692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10798.584378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10798.594370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10798.609781] Lustre: Skipped 1 previous similar message [10798.620864] Lustre: Skipped 1 previous similar message [10803.681905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10803.687361] Lustre: Skipped 1 previous similar message [10808.290152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10808.476094] Lustre: server umount lustre-OST0000 complete [10813.920547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10813.936377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10813.942867] Lustre: Skipped 1 previous similar message [10819.115206] Lustre: server umount lustre-MDT0000 complete [10822.773361] LustreError: 351813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780461052 with bad export cookie 13660022363906365624 [10822.781244] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10822.790705] LustreError: 351813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10823.380151] Lustre: server umount lustre-MDT0001 complete [10832.423579] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10833.796874] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 00:31:02 (1780461062) [10840.228579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10840.652208] LustreError: 355876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10840.676582] LustreError: 355876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10840.722260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10845.727360] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10846.181149] LustreError: 355877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10850.281469] LustreError: 355876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10854.205115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10854.485387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10859.307321] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10865.607494] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10870.322691] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10879.364488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10879.852555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10886.057451] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10891.590448] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10898.605824] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10898.705305] Lustre: lustre-OST0001: new disk, initializing [10898.712155] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10898.718775] Lustre: 358373:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10898.802268] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10904.767391] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10908.756977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10908.773996] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10908.854768] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10912.525936] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10919.217967] Lustre: lustre-OST0001: Not available for connect from 192.168.203.54@tcp (stopping) [10919.395189] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10919.415036] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10919.417968] LustreError: Skipped 1 previous similar message [10919.438987] Lustre: Skipped 2 previous similar messages [10922.315284] Lustre: server umount lustre-OST0001 complete [10924.274274] LustreError: 357473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10929.636977] LustreError: 357472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10929.651479] LustreError: 357472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10939.873781] LustreError: 357473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10939.876618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10939.879245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10939.879253] Lustre: Skipped 2 previous similar messages [10939.891602] LustreError: 357473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10939.903703] Lustre: Skipped 1 previous similar message [10950.118972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10950.132745] Lustre: Skipped 4 previous similar messages [10953.183119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10953.406467] Lustre: server umount lustre-OST0000 complete [10960.362791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10960.392537] Lustre: Skipped 1 previous similar message [10963.332314] Lustre: server umount lustre-MDT0000 complete [10965.475557] LustreError: 356832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10965.496425] LustreError: 356832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10967.542504] LustreError: 358372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780461197 with bad export cookie 13660022363906366506 [10967.551356] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10967.559038] LustreError: 358372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10974.067624] Lustre: server umount lustre-MDT0001 complete [10984.512622] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [10987.594809] Key type lgssc unregistered [10987.910787] LNet: 360241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10987.921255] LNetError: 360241:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10987.953514] LNet: Removed LNI 192.168.203.154@tcp [10988.700995] Key type .llcrypt unregistered [10988.703563] Key type ._llcrypt unregistered [11047.742132] Key type ._llcrypt registered [11047.744481] Key type .llcrypt registered [11047.881730] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [11049.291937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11049.361918] alg: No test for adler32 (adler32-zlib) [11050.430469] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [11050.664264] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [11052.407189] Key type lgssc registered [11053.728593] Lustre: Echo OBD driver; http://www.lustre.org/ [11061.951534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11061.996988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11063.301279] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11063.330269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11063.539773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11067.841546] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11076.072715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11076.195036] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11076.240769] Lustre: 363025:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11076.460919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11080.604499] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11086.235368] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11089.857830] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11091.937548] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11091.952718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11091.971781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11094.009956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11094.011877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11097.056911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11097.066451] Lustre: Skipped 2 previous similar messages [11102.184420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11102.193611] Lustre: Skipped 1 previous similar message [11106.272732] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11106.542957] Lustre: server umount lustre-MDT0000 complete [11110.265371] LustreError: 363017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780461340 with bad export cookie 3448005641811334412 [11110.268189] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11110.273648] LustreError: 363017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11110.636745] Lustre: server umount lustre-MDT0001 complete [11119.905839] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 00:35:48 (1780461348) [11126.148394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11126.653957] LustreError: 365311:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11126.675373] LustreError: 365311:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11126.722923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11130.927671] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11131.873147] LustreError: 365312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11137.004133] LustreError: 365311:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11139.314032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11139.653915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11143.894747] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11149.574306] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11153.824437] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11160.359631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11160.715859] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11160.928208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11167.613674] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11176.414054] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11186.547260] Lustre: Permanently deactivating lustre-OST0000 [11186.556879] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11192.398867] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11195.362581] LustreError: 367905:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11195.372782] LustreError: 367905:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11195.379877] LustreError: 367905:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11198.818633] LustreError: 367954:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11198.823979] LustreError: 367954:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11203.575773] Lustre: server umount lustre-OST0000 complete [11206.352828] LustreError: 368204:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11206.359495] LustreError: 368204:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11208.672735] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11208.687935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11208.692866] Lustre: Skipped 1 previous similar message [11208.712302] Lustre: Skipped 1 previous similar message [11218.915363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11218.929634] Lustre: Skipped 3 previous similar messages [11218.991404] Lustre: server umount lustre-MDT0000 complete [11222.803694] LustreError: 365292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780461452 with bad export cookie 3448005641811334986 [11222.812506] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11222.824486] LustreError: 365292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11232.388259] Lustre: server umount lustre-MDT0001 complete [11242.030271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11246.535188] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11248.994771] LustreError: 369346:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11255.401529] Lustre: 369686:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.154@tcp [11257.598092] Lustre: server umount MGS complete [11269.410545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11269.789700] LustreError: 370411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11269.811832] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11269.823861] Lustre: Skipped 1 previous similar message [11269.879412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11274.271667] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11275.233965] LustreError: 370412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11282.813178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11283.065095] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11287.675919] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11292.883649] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11296.528094] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11298.288738] Lustre: Permanently reactivating lustre-OST0000 [11298.309968] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11298.314970] Lustre: Skipped 2 previous similar messages [11325.924639] LustreError: 371965:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11332.963674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11333.324176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11333.333677] Lustre: Skipped 1 previous similar message [11339.786567] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11346.292913] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11353.708913] Lustre: server umount lustre-OST0000 complete [11360.228191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11360.238285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11360.256146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11367.520389] Lustre: server umount lustre-MDT0000 complete [11371.514320] LustreError: 371123:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11371.544406] LustreError: 371123:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11371.851861] LustreError: 370392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780461601 with bad export cookie 3448005641811336043 [11371.864208] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11371.866216] LustreError: 370392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11381.449671] Lustre: server umount lustre-MDT0001 complete [11393.428946] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [11396.323315] Key type lgssc unregistered [11396.654110] LNet: 374420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11396.663513] LNetError: 374420:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11396.680598] LNet: Removed LNI 192.168.203.154@tcp [11397.518476] Key type .llcrypt unregistered [11397.521262] Key type ._llcrypt unregistered [11417.390892] Key type ._llcrypt registered [11417.393881] Key type .llcrypt registered [11417.523875] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [11432.131898] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [11433.330876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11433.434656] alg: No test for adler32 (adler32-zlib) [11434.585722] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [11434.920860] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [11436.615137] Key type lgssc registered [11437.943919] Lustre: Echo OBD driver; http://www.lustre.org/ [11475.714795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11475.756980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11477.200793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11477.242918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11477.329477] Lustre: lustre-MDT0000: new disk, initializing [11477.490367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11477.526478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11482.132335] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11492.948613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11493.069888] Lustre: 377541:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11493.122947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11493.127802] Lustre: Skipped 1 previous similar message [11493.216541] Lustre: lustre-MDT0001: new disk, initializing [11493.348819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11493.446312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11493.473650] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11497.971382] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11505.321211] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11509.080708] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11515.897960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11516.189240] Lustre: lustre-OST0000: new disk, initializing [11516.193704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11516.199373] Lustre: 379283:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11516.298586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11522.176938] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11523.180099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11523.189108] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11523.355465] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11530.014395] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11534.887977] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11535.106128] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11539.657829] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11539.869169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11544.034476] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11544.043643] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11544.044908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11544.053738] LustreError: Skipped 1 previous similar message [11549.170227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11549.185026] Lustre: Skipped 1 previous similar message [11554.279565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11554.295471] Lustre: Skipped 1 previous similar message [11556.321117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11556.597296] Lustre: server umount lustre-OST0000 complete [11564.512980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11564.524922] Lustre: Skipped 1 previous similar message [11564.534542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11564.547662] Lustre: Skipped 1 previous similar message [11569.633037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11569.643044] Lustre: Skipped 1 previous similar message [11575.270747] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11575.574419] Lustre: server umount lustre-MDT0000 complete [11579.634750] LustreError: 379282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780461809 with bad export cookie 15687901818098948213 [11579.637837] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11579.650927] LustreError: 379282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11580.048334] Lustre: server umount lustre-MDT0001 complete [11589.407569] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11591.388976] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 00:43:39 (1780461819) [11598.781710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11599.394795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11603.881734] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11604.448786] LustreError: 381601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11604.470483] LustreError: 381601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11609.573882] LustreError: 381602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11612.224295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11612.473143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11616.591848] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11622.182976] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11626.101111] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11632.781141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11633.181727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11639.677569] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11644.347946] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11646.067185] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11648.921840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11657.707795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11657.728937] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11657.752505] Lustre: Skipped 1 previous similar message [11657.773092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11657.783314] Lustre: Skipped 2 previous similar messages [11659.746808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11661.550110] Lustre: server umount lustre-OST0000 complete [11668.961384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11668.973363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11672.531799] Lustre: server umount lustre-MDT0000 complete [11675.104781] LustreError: 381598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11675.132690] LustreError: 381598:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11676.475547] LustreError: 381581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780461906 with bad export cookie 15687901818098949095 [11676.484254] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11676.495046] LustreError: 381581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11676.942731] Lustre: server umount lustre-MDT0001 complete [11686.670106] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [11689.981619] Key type lgssc unregistered [11690.315991] LNet: 384967:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11690.323312] LNetError: 384967:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11690.341961] LNet: Removed LNI 192.168.203.154@tcp [11691.230373] Key type .llcrypt unregistered [11691.232253] Key type ._llcrypt unregistered [11712.722574] Key type ._llcrypt registered [11712.726327] Key type .llcrypt registered [11712.831801] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [11713.111378] Key type .llcrypt unregistered [11713.119624] Key type ._llcrypt unregistered [11733.879358] Key type ._llcrypt registered [11733.880690] Key type .llcrypt registered [11733.985456] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [11735.109653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11735.261596] alg: No test for adler32 (adler32-zlib) [11736.475274] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [11736.773671] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [11738.543399] Key type lgssc registered [11739.762751] Lustre: Echo OBD driver; http://www.lustre.org/ [11748.263530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11748.319404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11749.955809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11754.725188] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11759.078146] LustreError: 387146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11759.100591] LustreError: 387146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11764.196797] LustreError: 387147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11765.701196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11766.178959] LustreError: 387146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11766.215982] LustreError: 387146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11766.242496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11771.369866] LustreError: 387147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11771.394840] LustreError: 387147:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11773.745825] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11774.469805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11779.560533] LustreError: 387146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11779.574112] LustreError: 387146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11779.599421] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11786.800784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11787.210201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11792.194347] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11800.035755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11800.050798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11800.068479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11802.220475] Lustre: server umount lustre-OST0000 complete [11807.713305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11807.715091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11807.725957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11807.728106] LustreError: Skipped 1 previous similar message [11807.754518] Lustre: Skipped 1 previous similar message [11808.531546] Lustre: server umount lustre-MDT0000 complete [11812.446440] LustreError: 388759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780462042 with bad export cookie 2768385438411702853 [11812.451394] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11812.459069] LustreError: 388759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11822.214618] Lustre: server umount lustre-MDT0001 complete [11833.219225] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [11836.689672] Key type lgssc unregistered [11836.935671] LNet: 390130:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11836.940832] LNetError: 390130:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11836.957935] LNet: Removed LNI 192.168.203.154@tcp [11837.900248] Key type .llcrypt unregistered [11837.904832] Key type ._llcrypt unregistered [11861.131124] Key type ._llcrypt registered [11861.142608] Key type .llcrypt registered [11861.278661] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [11862.281077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11862.390937] alg: No test for adler32 (adler32-zlib) [11863.513871] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [11863.792673] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [11865.504612] Key type lgssc registered [11866.577658] Lustre: Echo OBD driver; http://www.lustre.org/ [11875.058640] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11875.113331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11890.847828] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11895.967213] LustreError: 391504:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.154@tcp: failed processing log, type 4: rc = -110 [11922.655229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11929.846324] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11936.191307] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11943.989697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11944.619206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11944.686578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11949.084165] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11950.054225] LustreError: 391529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11950.089285] LustreError: 391529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11955.183380] LustreError: 391530:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11957.229612] LustreError: 392427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11957.263406] LustreError: 392427:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11958.379612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11958.820057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11964.138759] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11970.252409] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11974.336732] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11983.818277] Lustre: server umount lustre-OST0000 complete [11989.474174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11989.476180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11989.504251] Lustre: Skipped 1 previous similar message [11989.522070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11994.601202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11994.611635] Lustre: Skipped 2 previous similar messages [11997.761667] Lustre: server umount lustre-MDT0000 complete [11999.712844] LustreError: 392433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11999.742547] LustreError: 392433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12000.079797] Lustre: 390841:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780462214/real 1780462214] req@00000000de131580 x1866949819658624/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1780462230 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12000.099235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12001.927737] LustreError: 391510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780462231 with bad export cookie 12968624719226282486 [12001.935885] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12001.942902] LustreError: 391510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12002.512773] Lustre: server umount lustre-MDT0001 complete [12014.952773] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [12017.777265] Key type lgssc unregistered [12018.067888] LNet: 394719:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12018.075749] LNetError: 394719:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12019.112366] LNet: Removed LNI 192.168.203.154@tcp [12019.915260] Key type .llcrypt unregistered [12019.924372] Key type ._llcrypt unregistered [12043.799608] Key type ._llcrypt registered [12043.807390] Key type .llcrypt registered [12043.910549] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [12045.121020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12045.193439] alg: No test for adler32 (adler32-zlib) [12046.312309] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [12046.533276] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [12048.255190] Key type lgssc registered [12049.420608] Lustre: Echo OBD driver; http://www.lustre.org/ [12057.439774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12057.456792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12058.908868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12063.076724] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12068.332375] LustreError: 396118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12068.349326] LustreError: 396118:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12071.499915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12071.929905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12076.214716] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12082.144141] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12086.350933] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12094.892608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12095.332860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12102.358939] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12103.576023] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12108.383766] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12113.899646] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12113.905315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12113.922605] Lustre: Skipped 1 previous similar message [12113.933542] Lustre: Skipped 2 previous similar messages [12116.243606] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.54@tcp (stopping) [12119.012133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12119.035130] Lustre: Skipped 2 previous similar messages [12119.903635] Lustre: server umount lustre-MDT0000 complete [12121.333739] LustreError: 396113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12122.927871] LustreError: 398357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.54@tcp arrived at 1780462352 with bad export cookie 14855955778280624372 [12122.962483] LustreError: 398357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12124.128925] LustreError: 396647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12124.164623] LustreError: 396647:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12125.089723] LustreError: 396099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780462355 with bad export cookie 14855955778280623476 [12125.114252] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12127.201383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12127.212090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12127.234060] Lustre: Skipped 1 previous similar message [12127.244225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12129.249590] LustreError: 396113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12129.270921] LustreError: 396113:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12131.541313] Lustre: server umount lustre-OST0000 complete [12135.393040] LustreError: 396119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12135.415087] LustreError: 396119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12138.848484] Lustre: server umount lustre-MDT0001 complete [12148.810274] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [12152.294777] Key type lgssc unregistered [12152.688644] LNet: 399447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12152.697943] LNetError: 399447:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12153.772381] LNet: Removed LNI 192.168.203.154@tcp [12154.996753] Key type .llcrypt unregistered [12154.999103] Key type ._llcrypt unregistered [12181.537432] Key type ._llcrypt registered [12181.540405] Key type .llcrypt registered [12181.666407] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [12183.000764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12183.025218] alg: No test for adler32 (adler32-zlib) [12184.142768] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [12184.487104] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [12186.223182] Key type lgssc registered [12187.668643] Lustre: Echo OBD driver; http://www.lustre.org/ [12197.235745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12197.247762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12198.956749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12203.651972] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12208.117287] LustreError: 400848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12208.165419] LustreError: 400848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12213.232213] LustreError: 400849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12214.252849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12214.712671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12221.321416] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12231.585432] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12236.265491] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12244.686330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12245.215780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12252.936174] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12259.477621] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12268.785030] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12276.034068] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.54@tcp (stopping) [12276.192195] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12276.207992] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12279.272051] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12279.272683] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12279.283579] Lustre: Skipped 1 previous similar message [12280.497075] Lustre: server umount lustre-MDT0001 complete [12281.089631] LustreError: 402058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12284.402771] LustreError: 400845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12284.423553] LustreError: 400845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12289.507135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12289.515168] LustreError: 400849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12289.520663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12289.526199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12289.575321] Lustre: Skipped 1 previous similar message [12292.462685] Lustre: server umount lustre-OST0000 complete [12297.390970] Lustre: server umount lustre-MDT0000 complete [12312.761866] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [12315.943256] Key type lgssc unregistered [12316.305493] LNet: 404202:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12316.316435] LNetError: 404202:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12316.348933] LNet: Removed LNI 192.168.203.154@tcp [12317.465395] Key type .llcrypt unregistered [12317.471280] Key type ._llcrypt unregistered [12346.463335] Key type ._llcrypt registered [12346.465427] Key type .llcrypt registered [12346.579855] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [12360.324815] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [12361.431768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12361.565848] alg: No test for adler32 (adler32-zlib) [12362.745664] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [12363.047269] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [12364.719197] Key type lgssc registered [12366.133692] Lustre: Echo OBD driver; http://www.lustre.org/ [12405.194417] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [12421.268241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12421.295929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12422.660947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12422.705414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12422.793586] Lustre: lustre-MDT0000: new disk, initializing [12422.908249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12422.934060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12427.486983] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12442.255490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12442.561663] Lustre: lustre-OST0000: new disk, initializing [12442.565802] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12442.573830] Lustre: Skipped 1 previous similar message [12442.581418] Lustre: 409170:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12442.685764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12448.374391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12448.380951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12448.418233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12448.962751] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12455.758610] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12470.628702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12470.808828] Lustre: 408167:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12470.957180] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12471.090966] Lustre: lustre-MDT0001: new disk, initializing [12471.232481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12471.273498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12471.288944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12476.068894] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12486.642122] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12486.814353] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12492.548728] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12492.871880] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12501.999156] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12502.006538] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12502.044577] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12502.061251] Lustre: Skipped 1 previous similar message [12502.083128] Lustre: Skipped 1 previous similar message [12503.871698] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.54@tcp (stopping) [12505.530575] Lustre: server umount lustre-MDT0001 complete [12512.235146] LustreError: 408175:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12512.268184] LustreError: 408175:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12517.377122] LustreError: 409179:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12517.427146] LustreError: 409179:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12522.466758] LustreError: 408180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12527.585635] LustreError: 411261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12527.589798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12527.622753] LustreError: 411261:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12527.665749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12531.692571] Lustre: server umount lustre-OST0000 complete [12532.718442] LustreError: 408180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12542.017602] Lustre: server umount lustre-MDT0000 complete [12559.223482] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [12562.772269] Key type lgssc unregistered [12563.182685] LNet: 412738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12563.206179] LNetError: 412738:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12563.225909] LNet: Removed LNI 192.168.203.154@tcp [12564.312243] Key type .llcrypt unregistered [12564.316027] Key type ._llcrypt unregistered [12609.301206] Key type ._llcrypt registered [12609.304200] Key type .llcrypt registered [12609.418235] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [12609.722387] Key type .llcrypt unregistered [12609.724614] Key type ._llcrypt unregistered [12627.239729] Key type ._llcrypt registered [12627.246609] Key type .llcrypt registered [12627.392534] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [12628.687313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12628.698746] alg: No test for adler32 (adler32-zlib) [12629.661693] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [12629.945858] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [12631.657065] Key type lgssc registered [12633.315715] Lustre: Echo OBD driver; http://www.lustre.org/ [12661.384093] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12661.419927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12662.725685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12662.821917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12662.952722] Lustre: lustre-MDT0000: new disk, initializing [12663.090587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12663.137949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12668.034741] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12678.335350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12678.453375] Lustre: 416098:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12678.542937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12678.550399] Lustre: Skipped 1 previous similar message [12678.608620] Lustre: lustre-MDT0001: new disk, initializing [12678.685024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12678.730298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12678.746522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12682.717522] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12691.007481] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12695.629603] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12702.420512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12702.696505] Lustre: lustre-OST0000: new disk, initializing [12702.709032] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12702.716785] Lustre: 417841:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12702.825258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12708.757598] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12712.510149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12712.525649] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12712.623821] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12716.164823] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12723.172229] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12723.178636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12723.200634] Lustre: Skipped 2 previous similar messages [12723.208387] Lustre: Skipped 1 previous similar message [12728.303468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12728.314258] Lustre: Skipped 3 previous similar messages [12729.222473] Lustre: server umount lustre-MDT0000 complete [12733.369540] LustreError: 416091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780462963 with bad export cookie 17730124055889554364 [12733.375341] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12733.380881] LustreError: 416091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12733.411821] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12733.429737] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12733.814804] Lustre: server umount lustre-MDT0001 complete [12740.404967] Lustre: server umount lustre-OST0000 complete [12753.189184] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 01:03:01 (1780462981) [12761.973876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12762.401440] LustreError: 419754:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12762.415059] LustreError: 419754:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12762.462102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12767.720957] LustreError: 419755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12767.813222] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12772.834101] LustreError: 419754:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12776.710899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12777.106532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12782.144799] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12788.187733] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12792.171870] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12811.887218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12812.076651] Lustre: Found index 0 for lustre-OST0000, updating log [12812.262175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12817.454848] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12823.050559] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12834.810937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12844.000147] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12844.007597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12844.029972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12850.292215] Lustre: server umount lustre-OST0000 complete [12869.088956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12869.103119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12869.130209] Lustre: Skipped 1 previous similar message [12869.137238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12869.148662] Lustre: Skipped 3 previous similar messages [12873.065607] Lustre: server umount lustre-MDT0000 complete [12876.947095] LustreError: 419734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780463106 with bad export cookie 17730124055889555757 [12876.960186] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12876.964660] LustreError: 419734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12877.417388] Lustre: server umount lustre-MDT0001 complete [12879.903193] LNet: 1 local NIs in recovery (showing 1): 192.168.203.154@tcp [12886.281862] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 01:05:15 (1780463115) [13511.061248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13511.608723] LustreError: 425860:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13511.637611] LustreError: 425860:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13511.703179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13516.682526] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13516.778484] LustreError: 425861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13521.895904] LustreError: 425860:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13526.125436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13526.566915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13530.678451] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13536.752505] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13540.440944] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13560.890187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13561.204718] Lustre: Found index 0 for lustre-OST0000, updating log [13561.610460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13567.963858] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13570.080559] LNet: 1 local NIs in recovery (showing 1): 192.168.203.154@tcp [13575.320882] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13578.668765] Lustre: server umount lustre-OST0000 complete [13583.490139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13583.495927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13583.503050] Lustre: Skipped 2 previous similar messages [13583.515456] Lustre: Skipped 5 previous similar messages [13584.551140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13584.560530] Lustre: Skipped 1 previous similar message [13586.592834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13586.603288] Lustre: Skipped 3 previous similar messages [13589.728851] LustreError: 426866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13589.749829] LustreError: 426866:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13589.788119] Lustre: server umount lustre-MDT0000 complete [13593.831888] LustreError: 425861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13593.857727] LustreError: 425861:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [13594.203703] LustreError: 425841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780463824 with bad export cookie 17730124055889556842 [13594.213994] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13594.225254] LustreError: 425841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13594.765752] Lustre: server umount lustre-MDT0001 complete [13624.555926] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [13634.149637] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [13634.594184] LNet: 1 local NIs in recovery (showing 1): 192.168.203.154@tcp [13671.512868] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 01:18:19 (1780463899) [13691.157952] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [13693.878439] Key type lgssc unregistered [13694.182633] LNet: 432075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13694.189034] LNetError: 432075:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13821.160260] LNet: Removed LNI 192.168.203.154@tcp [13822.224918] Key type .llcrypt unregistered [13822.226823] Key type ._llcrypt unregistered [13841.480384] Key type ._llcrypt registered [13841.484694] Key type .llcrypt registered [13841.596880] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [13842.795704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13842.891065] alg: No test for adler32 (adler32-zlib) [13843.982094] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [13844.250435] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [13845.927329] Key type lgssc registered [13847.298212] Lustre: Echo OBD driver; http://www.lustre.org/ [13852.636963] LNet: Added LNI 192.168.203.154@tcp100 [8/256/0/180] [13853.804578] LNet: Added LNI 192.168.203.154@tcp101 [8/256/0/180] [13855.104734] LNet: Added LNI 192.168.203.154@tcp102 [8/256/0/180] [13856.040243] LNet: Added LNI 192.168.203.154@tcp103 [8/256/0/180] [13857.215772] LNet: Added LNI 192.168.203.154@tcp104 [8/256/0/180] [13858.245948] LNet: Added LNI 192.168.203.154@tcp105 [8/256/0/180] [13859.356357] LNet: Added LNI 192.168.203.154@tcp106 [8/256/0/180] [13860.352840] LNet: Added LNI 192.168.203.154@tcp107 [8/256/0/180] [13861.455371] LNet: Added LNI 192.168.203.154@tcp108 [8/256/0/180] [13862.601103] LNet: Added LNI 192.168.203.154@tcp109 [8/256/0/180] [13863.594098] LNet: Added LNI 192.168.203.154@tcp110 [8/256/0/180] [13864.607796] LNet: Added LNI 192.168.203.154@tcp111 [8/256/0/180] [13865.593977] LNet: Added LNI 192.168.203.154@tcp112 [8/256/0/180] [13866.622957] LNet: Added LNI 192.168.203.154@tcp113 [8/256/0/180] [13867.621849] LNet: Added LNI 192.168.203.154@tcp114 [8/256/0/180] [13868.527733] LNet: Added LNI 192.168.203.154@tcp115 [8/256/0/180] [13869.509077] LNet: Added LNI 192.168.203.154@tcp116 [8/256/0/180] [13870.395684] LNet: Added LNI 192.168.203.154@tcp117 [8/256/0/180] [13871.307325] LNet: Added LNI 192.168.203.154@tcp118 [8/256/0/180] [13872.337141] LNet: Added LNI 192.168.203.154@tcp119 [8/256/0/180] [13873.224363] LNet: Added LNI 192.168.203.154@tcp120 [8/256/0/180] [13874.028143] LNet: Added LNI 192.168.203.154@tcp121 [8/256/0/180] [13874.944488] LNet: Added LNI 192.168.203.154@tcp122 [8/256/0/180] [13875.923297] LNet: Added LNI 192.168.203.154@tcp123 [8/256/0/180] [13876.925768] LNet: Added LNI 192.168.203.154@tcp124 [8/256/0/180] [13877.801262] LNet: Added LNI 192.168.203.154@tcp125 [8/256/0/180] [13878.675355] LNet: Added LNI 192.168.203.154@tcp126 [8/256/0/180] [13879.606136] LNet: Added LNI 192.168.203.154@tcp127 [8/256/0/180] [13880.430591] LNet: Added LNI 192.168.203.154@tcp128 [8/256/0/180] [13881.229882] LNet: Added LNI 192.168.203.154@tcp129 [8/256/0/180] [13881.994911] LNet: Added LNI 192.168.203.154@tcp130 [8/256/0/180] [13882.834957] LNet: Added LNI 192.168.203.154@tcp131 [8/256/0/180] [13883.647466] LNet: Added LNI 192.168.203.154@tcp132 [8/256/0/180] [13884.442328] LNet: Added LNI 192.168.203.154@tcp133 [8/256/0/180] [13885.357523] LNet: Added LNI 192.168.203.154@tcp134 [8/256/0/180] [13886.198317] LNet: Added LNI 192.168.203.154@tcp135 [8/256/0/180] [13893.526661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13893.546307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13894.698326] Lustre: 435095:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.154@tcp: can't add NID 192.168.203.154@tcp130: rc = -75 [13894.835658] LNetError: 435126:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [13894.849796] LNetError: Protocol error connecting to 192.168.203.154@tcp119 at host 192.168.203.154:988: is it running a compatible version of Lustre? [13894.859266] LNetError: Skipped 1 previous similar message [13895.020323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13895.067300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13895.186362] Lustre: lustre-MDT0000: new disk, initializing [13895.286347] Lustre: 435151:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.154@tcp: can't add NID 192.168.203.154@tcp130: rc = -75 [13895.294832] Lustre: 435151:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [13895.312532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13895.339463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13895.456109] LNet: 3 local NIs in recovery (showing 3): 192.168.203.154@tcp125, 192.168.203.154@tcp127, 192.168.203.154@tcp119 [13895.461644] LNetError: 432553:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [13895.492794] LNetError: 432553:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [13895.502497] LNetError: Protocol error connecting to 192.168.203.154@tcp103 at host 192.168.203.154:988: is it running a compatible version of Lustre? [13895.524456] LNetError: Skipped 6 previous similar messages [13897.514287] LNetError: 435125:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [13897.536545] LNetError: 435125:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [13897.549374] LNetError: Protocol error connecting to 192.168.203.154@tcp125 at host 192.168.203.154:988: is it running a compatible version of Lustre? [13900.427488] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13901.609356] LNetError: 435122:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [13901.637905] LNetError: Protocol error connecting to 192.168.203.154@tcp125 at host 192.168.203.154:988: is it running a compatible version of Lustre? [13909.792506] LNetError: 432553:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [13909.809805] LNetError: Protocol error connecting to 192.168.203.154@tcp125 at host 192.168.203.154:988: is it running a compatible version of Lustre? [13910.548723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13910.697881] Lustre: 435113:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13910.741124] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13910.744318] Lustre: Skipped 1 previous similar message [13910.794987] Lustre: lustre-MDT0001: new disk, initializing [13910.821836] Lustre: 435897:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.154@tcp: can't add NID 192.168.203.154@tcp130: rc = -75 [13910.828792] Lustre: 435897:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [13910.930241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13910.965659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13910.980198] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13914.887186] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13918.760397] Lustre: 436304:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.154@tcp: can't add NID 192.168.203.154@tcp130: rc = -75 [13918.771381] Lustre: 436304:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13922.057335] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13925.153942] LNetError: 435128:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [13925.172672] LNetError: Protocol error connecting to 192.168.203.154@tcp125 at host 192.168.203.154:988: is it running a compatible version of Lustre? [13926.287258] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13933.081253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13933.443229] Lustre: lustre-OST0000: new disk, initializing [13933.448615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13933.467749] Lustre: 436859:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13933.570677] Lustre: 436865:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.154@tcp: can't add NID 192.168.203.154@tcp130: rc = -75 [13933.578268] Lustre: 436865:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [13933.620478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13939.809366] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13941.809504] Lustre: 437299:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.154@tcp: can't add NID 192.168.203.154@tcp130: rc = -75 [13941.822974] Lustre: 437299:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13941.879474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13941.889027] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13942.012305] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13944.615063] LNetError: 435125:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [13944.624032] LNetError: 435125:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [13944.640775] LNetError: Protocol error connecting to 192.168.203.154@tcp123 at host 192.168.203.154:988: is it running a compatible version of Lustre? [13944.654104] LNetError: Skipped 6 previous similar messages [13947.387577] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13955.873813] LNet: 1 local NIs in recovery (showing 1): 192.168.203.154@tcp125 [13957.600432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13957.606921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13957.615752] LustreError: Skipped 1 previous similar message [13957.618979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13957.662933] Lustre: Skipped 1 previous similar message [13961.869573] Lustre: server umount lustre-OST0000 complete [13967.330434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13967.340800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13967.358745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13967.363160] Lustre: Skipped 1 previous similar message [13972.961227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13972.965155] Lustre: Skipped 2 previous similar messages [13977.059016] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13980.127488] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13980.419905] Lustre: server umount lustre-MDT0000 complete [13984.193105] LustreError: 435105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780464214 with bad export cookie 14335062866482627346 [13984.193790] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13984.203887] LustreError: 435105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13984.556850] Lustre: server umount lustre-MDT0001 complete [13987.616386] LNetError: 432554:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [13987.632346] LNetError: 432554:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 13 previous similar messages [13987.640215] LNetError: Protocol error connecting to 192.168.203.154@tcp103 at host 192.168.203.154:988: is it running a compatible version of Lustre? [13987.649919] LNetError: Skipped 13 previous similar messages [13996.306192] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [13999.618918] Key type lgssc unregistered [14000.142631] LNet: 438642:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14000.155993] LNetError: 438642:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14000.171222] LNet: Removed LNI 192.168.203.154@tcp [14001.183179] LNet: Removed LNI 192.168.203.154@tcp100 [14001.186482] LNetError: 438642:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14001.199465] LNetError: 438642:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14001.206444] LNet: Removed LNI 192.168.203.154@tcp101 [14002.273297] LNet: Removed LNI 192.168.203.154@tcp102 [14002.283049] LNetError: 438642:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14002.295800] LNetError: 438642:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14002.309702] LNet: Removed LNI 192.168.203.154@tcp103 [14003.359183] LNet: Removed LNI 192.168.203.154@tcp104 [14004.384120] LNet: Removed LNI 192.168.203.154@tcp105 [14004.389593] LNetError: 438642:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14004.397292] LNetError: 438642:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14004.409382] LNet: Removed LNI 192.168.203.154@tcp106 [14005.472276] LNet: Removed LNI 192.168.203.154@tcp107 [14006.496345] LNet: Removed LNI 192.168.203.154@tcp108 [14007.519710] LNet: Removed LNI 192.168.203.154@tcp109 [14008.545064] LNet: Removed LNI 192.168.203.154@tcp110 [14008.547325] LNetError: 438642:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14008.558934] LNetError: 438642:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14008.567839] LNet: Removed LNI 192.168.203.154@tcp111 [14009.633566] LNet: Removed LNI 192.168.203.154@tcp112 [14010.655186] LNet: Removed LNI 192.168.203.154@tcp113 [14011.679951] LNet: Removed LNI 192.168.203.154@tcp114 [14012.703776] LNet: Removed LNI 192.168.203.154@tcp115 [14013.727194] LNet: Removed LNI 192.168.203.154@tcp116 [14014.757042] LNet: Removed LNI 192.168.203.154@tcp117 [14015.780268] LNet: Removed LNI 192.168.203.154@tcp118 [14016.799336] LNet: Removed LNI 192.168.203.154@tcp119 [14016.802926] LNetError: 438642:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14016.811606] LNetError: 438642:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14016.821581] LNet: Removed LNI 192.168.203.154@tcp120 [14017.887485] LNet: Removed LNI 192.168.203.154@tcp121 [14018.911515] LNet: Removed LNI 192.168.203.154@tcp122 [14019.935146] LNet: Removed LNI 192.168.203.154@tcp123 [14020.961135] LNet: Removed LNI 192.168.203.154@tcp124 [14020.963133] LNet: Removed LNI 192.168.203.154@tcp125 [14021.983160] LNet: Removed LNI 192.168.203.154@tcp126 [14023.007173] LNet: Removed LNI 192.168.203.154@tcp127 [14024.031218] LNet: Removed LNI 192.168.203.154@tcp128 [14025.055231] LNet: Removed LNI 192.168.203.154@tcp129 [14025.059956] LNet: Removed LNI 192.168.203.154@tcp130 [14025.065977] LNet: Removed LNI 192.168.203.154@tcp131 [14025.081716] LNet: Removed LNI 192.168.203.154@tcp132 [14025.087584] LNet: Removed LNI 192.168.203.154@tcp133 [14025.094130] LNet: Removed LNI 192.168.203.154@tcp134 [14025.110402] LNet: Removed LNI 192.168.203.154@tcp135 [14025.913476] Key type .llcrypt unregistered [14025.916301] Key type ._llcrypt unregistered [14049.468651] Key type ._llcrypt registered [14049.477237] Key type .llcrypt registered [14049.585108] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [14063.363220] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [14064.215927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14064.273533] alg: No test for adler32 (adler32-zlib) [14065.361911] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [14065.573130] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [14067.231136] Key type lgssc registered [14068.248041] Lustre: Echo OBD driver; http://www.lustre.org/ [14101.772915] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 01:25:30 (1780464330) [14117.891674] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [14120.346898] Key type lgssc unregistered [14120.732821] LNet: 442677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14120.742597] LNetError: 442677:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14120.770214] LNet: Removed LNI 192.168.203.154@tcp [14121.558821] Key type .llcrypt unregistered [14121.561862] Key type ._llcrypt unregistered [14135.242890] Key type ._llcrypt registered [14135.245505] Key type .llcrypt registered [14135.357765] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [14136.158793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14136.191435] alg: No test for adler32 (adler32-zlib) [14137.183508] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [14137.373620] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [14139.023172] Key type lgssc registered [14139.992245] Lustre: Echo OBD driver; http://www.lustre.org/ [14148.317769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14148.343971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14149.570702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14149.602330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14149.696435] Lustre: lustre-MDT0000: new disk, initializing [14149.813709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14149.843422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14154.134314] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14164.492794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14164.599713] Lustre: 443853:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14164.660382] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14164.663865] Lustre: Skipped 1 previous similar message [14164.735705] Lustre: lustre-MDT0001: new disk, initializing [14164.852870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14164.893918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14164.908077] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14169.549672] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14176.664842] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14180.647471] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14187.606410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14187.857920] Lustre: lustre-OST0000: new disk, initializing [14187.861340] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14187.867330] Lustre: 445595:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14187.939876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14195.062674] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14196.826339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14196.838370] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14196.929649] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14202.703058] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14206.762148] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing lnet_if_list [14212.690323] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14212.735364] LNet: Added LNI 192.168.203.154@tcp10 [8/256/0/180] [14218.104452] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14218.157066] LNet: Added LNI 192.168.203.154@tcp11 [8/256/0/180] [14222.474932] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14222.509553] LNet: Added LNI 192.168.203.154@tcp12 [8/256/0/180] [14228.022041] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14228.061050] LNet: Added LNI 192.168.203.154@tcp13 [8/256/0/180] [14232.980334] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14233.007955] LNet: Added LNI 192.168.203.154@tcp14 [8/256/0/180] [14237.778734] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14237.806513] LNet: Added LNI 192.168.203.154@tcp15 [8/256/0/180] [14242.408947] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14242.490333] LNet: Added LNI 192.168.203.154@tcp16 [8/256/0/180] [14247.211251] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14247.239233] LNet: Added LNI 192.168.203.154@tcp17 [8/256/0/180] [14252.749316] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14252.794801] LNet: Added LNI 192.168.203.154@tcp18 [8/256/0/180] [14257.995775] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14258.038036] LNet: Added LNI 192.168.203.154@tcp19 [8/256/0/180] [14262.809805] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14262.836489] LNet: Added LNI 192.168.203.154@tcp20 [8/256/0/180] [14268.335982] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14268.378991] LNet: Added LNI 192.168.203.154@tcp21 [8/256/0/180] [14273.537670] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14273.583495] LNet: Added LNI 192.168.203.154@tcp22 [8/256/0/180] [14278.898049] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14278.952134] LNet: Added LNI 192.168.203.154@tcp23 [8/256/0/180] [14284.064461] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14284.094230] LNet: Added LNI 192.168.203.154@tcp24 [8/256/0/180] [14289.240417] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14289.275809] LNet: Added LNI 192.168.203.154@tcp25 [8/256/0/180] [14294.600266] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14294.636926] LNet: Added LNI 192.168.203.154@tcp26 [8/256/0/180] [14300.134546] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14300.158211] LNet: Added LNI 192.168.203.154@tcp27 [8/256/0/180] [14305.045250] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14305.097292] LNet: Added LNI 192.168.203.154@tcp28 [8/256/0/180] [14310.600981] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14310.634638] LNet: Added LNI 192.168.203.154@tcp29 [8/256/0/180] [14316.376390] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14316.397158] LNet: Added LNI 192.168.203.154@tcp30 [8/256/0/180] [14321.791401] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14321.828578] LNet: Added LNI 192.168.203.154@tcp31 [8/256/0/180] [14327.448660] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14327.473795] LNet: Added LNI 192.168.203.154@tcp32 [8/256/0/180] [14332.937459] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14332.959439] LNet: Added LNI 192.168.203.154@tcp33 [8/256/0/180] [14338.832865] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14338.864298] LNet: Added LNI 192.168.203.154@tcp34 [8/256/0/180] [14345.047879] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14345.075628] LNet: Added LNI 192.168.203.154@tcp35 [8/256/0/180] [14350.581313] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14350.626581] LNet: Added LNI 192.168.203.154@tcp36 [8/256/0/180] [14356.484572] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14356.509095] LNet: Added LNI 192.168.203.154@tcp37 [8/256/0/180] [14362.962521] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14363.009367] LNet: Added LNI 192.168.203.154@tcp38 [8/256/0/180] [14369.820660] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14369.841743] LNet: Added LNI 192.168.203.154@tcp39 [8/256/0/180] [14377.195956] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14377.218432] LNet: Added LNI 192.168.203.154@tcp40 [8/256/0/180] [14383.909929] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14383.936361] LNet: Added LNI 192.168.203.154@tcp41 [8/256/0/180] [14390.567535] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14390.605602] LNet: Added LNI 192.168.203.154@tcp42 [8/256/0/180] [14397.346192] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14397.369614] LNet: Added LNI 192.168.203.154@tcp43 [8/256/0/180] [14403.716754] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14403.744456] LNet: Added LNI 192.168.203.154@tcp44 [8/256/0/180] [14409.502236] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14409.539289] LNet: Added LNI 192.168.203.154@tcp45 [8/256/0/180] [14415.749532] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14415.786438] LNet: Added LNI 192.168.203.154@tcp46 [8/256/0/180] [14422.792503] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14422.820025] LNet: Added LNI 192.168.203.154@tcp47 [8/256/0/180] [14430.065887] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14430.094157] LNet: Added LNI 192.168.203.154@tcp48 [8/256/0/180] [14437.692858] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14437.721981] LNet: Added LNI 192.168.203.154@tcp49 [8/256/0/180] [14445.267669] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14445.294427] LNet: Added LNI 192.168.203.154@tcp50 [8/256/0/180] [14461.927749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14461.937421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14461.951858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14463.457983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14463.462769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14463.485949] Lustre: Skipped 1 previous similar message [14466.906159] Lustre: server umount lustre-MDT0000 complete [14470.144283] LustreError: 443846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780464700 with bad export cookie 882158412406083456 [14470.151153] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14470.156319] LustreError: 443846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14470.381745] Lustre: server umount lustre-MDT0001 complete [14486.732042] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [14489.122724] Lustre: 443123:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780464703/real 1780464703] req@0000000019776b60 x1866952204248064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780464719 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14489.156862] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14491.103341] Lustre: 443124:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780464705/real 1780464705] req@0000000005448e99 x1866952204248192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780464721 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14494.175329] Lustre: 443125:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780464708/real 1780464708] req@00000000a1ed7122 x1866952204248320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780464724 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14494.801391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14495.330148] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc3e0e66f7165990 [14495.346525] Lustre: MGC192.168.203.154@tcp: Connection restored to 0@lo (at 0@lo) [14495.597452] LustreError: 445589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14495.635420] LustreError: 445589:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [14495.674888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14496.287134] Lustre: 443125:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780464710/real 1780464710] req@00000000f5d23263 x1866952204248576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780464726 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14498.722497] LustreError: 445590:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14499.273393] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14499.749074] LustreError: 468975:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14499.764407] LustreError: 468975:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14501.791980] Lustre: 443124:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780464715/real 1780464715] req@0000000093cd2ea4 x1866952204248704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780464731 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14504.934529] LustreError: 445589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14504.954924] LustreError: 445589:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14506.600732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14507.026615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14510.984621] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14512.108800] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14516.160832] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14518.251747] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14519.687748] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14536.673305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14536.681673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14536.700110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14536.705908] Lustre: Skipped 2 previous similar messages [14537.697053] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14538.722552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14538.730151] Lustre: Skipped 2 previous similar messages [14541.381140] Lustre: server umount lustre-OST0000 complete [14547.936947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14547.948158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14547.970065] Lustre: Skipped 1 previous similar message [14547.982440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14559.200220] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14559.204383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14559.227309] Lustre: Skipped 5 previous similar messages [14559.409954] Lustre: server umount lustre-MDT0000 complete [14562.977731] LustreError: 443846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780464792 with bad export cookie 882158412406086032 [14562.984083] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14562.988449] LustreError: 443846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14563.425422] Lustre: server umount lustre-MDT0001 complete [14573.353821] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [14575.714249] Key type lgssc unregistered [14576.102882] LNet: 471518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14576.113155] LNetError: 471518:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14576.122257] LNet: Removed LNI 192.168.203.154@tcp [14576.128040] LNet: Removed LNI 192.168.203.154@tcp10 [14576.131889] LNet: Removed LNI 192.168.203.154@tcp11 [14576.135658] LNet: Removed LNI 192.168.203.154@tcp12 [14576.138771] LNet: Removed LNI 192.168.203.154@tcp13 [14576.144392] LNet: Removed LNI 192.168.203.154@tcp14 [14576.146580] LNet: Removed LNI 192.168.203.154@tcp15 [14576.154343] LNet: Removed LNI 192.168.203.154@tcp16 [14576.158637] LNet: Removed LNI 192.168.203.154@tcp17 [14576.162262] LNet: Removed LNI 192.168.203.154@tcp18 [14576.167385] LNet: Removed LNI 192.168.203.154@tcp19 [14576.173842] LNet: Removed LNI 192.168.203.154@tcp20 [14576.177641] LNet: Removed LNI 192.168.203.154@tcp21 [14576.181189] LNet: Removed LNI 192.168.203.154@tcp22 [14576.183607] LNet: Removed LNI 192.168.203.154@tcp23 [14576.186771] LNet: Removed LNI 192.168.203.154@tcp24 [14576.192398] LNet: Removed LNI 192.168.203.154@tcp25 [14576.198163] LNet: Removed LNI 192.168.203.154@tcp26 [14576.201096] LNet: Removed LNI 192.168.203.154@tcp27 [14576.205596] LNet: Removed LNI 192.168.203.154@tcp28 [14576.209623] LNet: Removed LNI 192.168.203.154@tcp29 [14576.216083] LNet: Removed LNI 192.168.203.154@tcp30 [14576.219283] LNet: Removed LNI 192.168.203.154@tcp31 [14576.223279] LNet: Removed LNI 192.168.203.154@tcp32 [14576.227405] LNet: Removed LNI 192.168.203.154@tcp33 [14576.230433] LNet: Removed LNI 192.168.203.154@tcp34 [14576.234550] LNet: Removed LNI 192.168.203.154@tcp35 [14576.237770] LNet: Removed LNI 192.168.203.154@tcp36 [14576.240222] LNet: Removed LNI 192.168.203.154@tcp37 [14576.242682] LNet: Removed LNI 192.168.203.154@tcp38 [14576.246499] LNet: Removed LNI 192.168.203.154@tcp39 [14576.249904] LNet: Removed LNI 192.168.203.154@tcp40 [14576.254478] LNet: Removed LNI 192.168.203.154@tcp41 [14576.257344] LNet: Removed LNI 192.168.203.154@tcp42 [14576.260606] LNet: Removed LNI 192.168.203.154@tcp43 [14576.263303] LNet: Removed LNI 192.168.203.154@tcp44 [14576.265938] LNet: Removed LNI 192.168.203.154@tcp45 [14576.272945] LNet: Removed LNI 192.168.203.154@tcp46 [14576.275917] LNet: Removed LNI 192.168.203.154@tcp47 [14576.282234] LNet: Removed LNI 192.168.203.154@tcp48 [14576.284777] LNet: Removed LNI 192.168.203.154@tcp49 [14576.293944] LNet: Removed LNI 192.168.203.154@tcp50 [14576.848421] Key type .llcrypt unregistered [14576.850458] Key type ._llcrypt unregistered [14593.458957] Key type ._llcrypt registered [14593.461071] Key type .llcrypt registered [14593.560882] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [14604.947597] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [14605.780754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14605.861026] alg: No test for adler32 (adler32-zlib) [14606.861469] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [14607.020910] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [14608.671187] Key type lgssc registered [14609.509429] Lustre: Echo OBD driver; http://www.lustre.org/ [14637.240976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14637.285728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14638.480612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14638.526232] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14638.591168] Lustre: lustre-MDT0000: new disk, initializing [14638.666439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14638.687821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14642.044747] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14650.797790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14650.906728] Lustre: 474659:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14650.961263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14650.967218] Lustre: Skipped 1 previous similar message [14651.033781] Lustre: lustre-MDT0001: new disk, initializing [14651.107076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14651.151054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14651.169083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14654.568688] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14660.393220] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14663.481442] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14669.663788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14669.832703] Lustre: *** cfs_fail_loc=90f, val=0*** [14669.901932] Lustre: lustre-OST0000: new disk, initializing [14669.906478] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14669.912958] Lustre: 476449:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14669.998116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14674.882358] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14677.049939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14677.059180] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14677.170448] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14681.520684] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14687.715407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14687.716529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14687.717422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14687.732599] Lustre: Skipped 1 previous similar message [14687.745707] LustreError: Skipped 1 previous similar message [14692.856793] Lustre: server umount lustre-OST0000 complete [14697.440544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14697.445617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14697.458805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14697.461778] Lustre: Skipped 1 previous similar message [14703.073683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14703.084167] Lustre: Skipped 2 previous similar messages [14707.169741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14707.181135] Lustre: Skipped 1 previous similar message [14710.239356] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14710.358818] Lustre: server umount lustre-MDT0000 complete [14713.523273] LustreError: 476448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780464943 with bad export cookie 17425732703589346122 [14713.524590] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14713.530392] LustreError: 476448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14713.817598] Lustre: server umount lustre-MDT0001 complete [14721.414777] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [14723.368280] Key type lgssc unregistered [14723.613578] LNet: 478224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14723.622022] LNetError: 478224:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14723.646836] LNet: Removed LNI 192.168.203.154@tcp [14724.294302] Key type .llcrypt unregistered [14724.301590] Key type ._llcrypt unregistered [14737.574650] Key type ._llcrypt registered [14737.576500] Key type .llcrypt registered [14737.643670] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [14738.517356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14738.537087] alg: No test for adler32 (adler32-zlib) [14739.493808] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [14739.720805] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [14741.407166] Key type lgssc registered [14742.288814] Lustre: Echo OBD driver; http://www.lustre.org/ [14749.062075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14749.087306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14750.631249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14754.052122] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14760.942089] LustreError: 479454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14760.969489] LustreError: 479454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14760.997993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14761.266878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14764.720542] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14769.607910] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14772.628947] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14778.327645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14778.701583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14783.301947] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14787.214073] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14804.523642] loop0: detected capacity change from 0 to 204800000 [14804.626692] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14804.644159] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14804.658704] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14809.444647] loop0: detected capacity change from 0 to 204800000 [14809.603979] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14809.611224] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14809.624759] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14813.615071] loop0: detected capacity change from 0 to 204800000 [14818.038262] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14818.168375] Lustre: fs15246-OST0000: new disk, initializing [14818.173703] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14818.186753] Lustre: 482727:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14818.231322] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14822.823180] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14830.019307] loop1: detected capacity change from 0 to 204800000 [14833.955637] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14834.026723] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14834.063469] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14834.119656] Lustre: fs15246-MDT0000: new disk, initializing [14834.223821] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14834.247304] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14834.332026] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14835.360239] Lustre: 483835:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000f8b2ce10 x1866952835375744/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1780465081 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14837.245531] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14844.453375] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14844.464471] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14844.482635] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14857.711329] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14857.718859] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14859.680208] Lustre: server umount fs15246-MDT0000 complete [14862.822615] LustreError: 479449:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14866.189659] Lustre: server umount fs15246-OST0000 complete [14877.152427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14877.162346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14877.172959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14878.178709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14878.179230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14880.720919] Lustre: server umount lustre-OST0000 complete [14884.322627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14884.330312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14884.343406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14884.348293] Lustre: Skipped 1 previous similar message [14888.416576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14888.420066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14889.886680] Lustre: server umount lustre-MDT0000 complete [14892.549291] LustreError: 479434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780465122 with bad export cookie 12926243614985357724 [14892.551586] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14892.561693] LustreError: 479434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14892.799402] Lustre: server umount lustre-MDT0001 complete [14900.378761] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [14902.241282] Key type lgssc unregistered [14902.455790] LNet: 486895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14902.460878] LNetError: 486895:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14902.474600] LNet: Removed LNI 192.168.203.154@tcp [14903.010384] Key type .llcrypt unregistered [14903.012988] Key type ._llcrypt unregistered [14915.283626] Key type ._llcrypt registered [14915.285352] Key type .llcrypt registered [14915.356463] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [14915.875041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14915.904645] alg: No test for adler32 (adler32-zlib) [14916.866827] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [14917.016666] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [14918.623187] Key type lgssc registered [14919.497698] Lustre: Echo OBD driver; http://www.lustre.org/ [14944.194415] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [14949.975793] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [14972.876213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14972.897432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14974.082539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14974.121293] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14974.190541] Lustre: lustre-MDT0000: new disk, initializing [14974.300739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14974.326272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14977.476921] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14985.043614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14985.124913] Lustre: 491282:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14985.169931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14985.176149] Lustre: Skipped 1 previous similar message [14985.225176] Lustre: lustre-MDT0001: new disk, initializing [14985.297169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14985.322112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14985.331651] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14988.343370] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14994.013141] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14996.681142] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15001.378476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15001.591751] Lustre: lustre-OST0000: new disk, initializing [15001.594955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15001.599442] Lustre: 493024:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15001.666899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15005.472175] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [15010.406254] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15010.863683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15010.867925] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15010.957043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15013.003667] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15013.116937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15015.785959] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15015.922427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15021.026539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15021.030992] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15021.037164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15021.041050] Lustre: Skipped 1 previous similar message [15021.056768] Lustre: Skipped 1 previous similar message [15023.306948] Lustre: server umount lustre-OST0000 complete [15026.144878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15026.150211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15026.153714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15026.159136] Lustre: Skipped 1 previous similar message [15031.266708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15031.271662] Lustre: Skipped 2 previous similar messages [15031.933233] Lustre: server umount lustre-MDT0000 complete [15034.471020] LustreError: 491276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780465264 with bad export cookie 3667834455585372856 [15034.477789] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15034.478266] LustreError: 491276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15034.741692] Lustre: server umount lustre-MDT0001 complete [15049.544485] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [15055.100396] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [15072.788374] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14786 sec ========= 01:41:42 (1780465302) [15073.801562] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:41:43 (1780465303) === [15075.743563] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:41:44 (1780465304) === [15098.461433] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [15100.142358] Key type lgssc unregistered [15100.321267] LNet: 499458:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15100.327969] LNetError: 499458:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15100.342211] LNet: Removed LNI 192.168.203.154@tcp [15100.875762] Key type .llcrypt unregistered [15100.877754] Key type ._llcrypt unregistered