[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 473799985 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003217] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008323] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010015] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011011] pid_max: default: 32768 minimum: 301 [ 0.012137] LSM: Security Framework initializing [ 0.013052] Yama: becoming mindful. [ 0.014040] SELinux: Initializing. [ 0.015084] *** VALIDATE selinux *** [ 0.023786] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028574] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029134] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031101] *** VALIDATE tmpfs *** [ 0.032442] *** VALIDATE proc *** [ 0.033232] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035235] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036152] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038032] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.041626] debug: unmapping init [mem 0xffffffff99059000-0xffffffff99060fff] [ 0.044164] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045690] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046022] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053314] rcu: Hierarchical SRCU implementation. [ 0.055414] smp: Bringing up secondary CPUs ... [ 0.056614] x86: Booting SMP configuration: [ 0.057025] .... node #0, CPUs: #1 #2 #3 [ 0.061014] smp: Brought up 1 node, 4 CPUs [ 0.063019] smpboot: Max logical packages: 1 [ 0.064011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.153017] node 0 deferred pages initialised in 85ms [ 0.155123] devtmpfs: initialized [ 0.156216] x86/mm: Memory block size: 128MB [ 0.158803] gcov: version magic: 0x41383552 [ 0.160275] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162324] pinctrl core: initialized pinctrl subsystem [ 0.163194] [ 0.163868] ************************************************************* [ 0.164012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165014] ** ** [ 0.166012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167014] ** ** [ 0.168015] ** This means that this kernel is built to expose internal ** [ 0.169012] ** IOMMU data structures, which may compromise security on ** [ 0.170014] ** your system. ** [ 0.171013] ** ** [ 0.172014] ** If you see this message and you are not debugging the ** [ 0.173011] ** kernel, report this immediately to your vendor! ** [ 0.174014] ** ** [ 0.175014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176014] ************************************************************* [ 0.177668] NET: Registered protocol family 16 [ 0.178418] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.179070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.180060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.182030] cpuidle: using governor menu [ 0.184062] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.186464] PCI: Using configuration type 1 for base access [ 0.188124] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.198054] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.199069] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.200126] cryptd: max_cpu_qlen set to 1000 [ 0.202252] ACPI: Added _OSI(Module Device) [ 0.203014] ACPI: Added _OSI(Processor Device) [ 0.204013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.205012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.209431] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.211543] ACPI: Interpreter enabled [ 0.212137] ACPI: PM: (supports S0 S3 S4 S5) [ 0.213011] ACPI: Using IOAPIC for interrupt routing [ 0.214093] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.215414] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.225528] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.227020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.228072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.229912] acpiphp: Slot [2] registered [ 0.230067] acpiphp: Slot [5] registered [ 0.230999] acpiphp: Slot [6] registered [ 0.231105] acpiphp: Slot [7] registered [ 0.232120] acpiphp: Slot [8] registered [ 0.233109] acpiphp: Slot [9] registered [ 0.234117] acpiphp: Slot [10] registered [ 0.235145] acpiphp: Slot [3] registered [ 0.236118] acpiphp: Slot [4] registered [ 0.237089] acpiphp: Slot [11] registered [ 0.238099] acpiphp: Slot [12] registered [ 0.239113] acpiphp: Slot [13] registered [ 0.240102] acpiphp: Slot [14] registered [ 0.241092] acpiphp: Slot [15] registered [ 0.242099] acpiphp: Slot [16] registered [ 0.243096] acpiphp: Slot [17] registered [ 0.244089] acpiphp: Slot [18] registered [ 0.245117] acpiphp: Slot [19] registered [ 0.246112] acpiphp: Slot [20] registered [ 0.247096] acpiphp: Slot [21] registered [ 0.248103] acpiphp: Slot [22] registered [ 0.249089] acpiphp: Slot [23] registered [ 0.250097] acpiphp: Slot [24] registered [ 0.251148] acpiphp: Slot [25] registered [ 0.252097] acpiphp: Slot [26] registered [ 0.253113] acpiphp: Slot [27] registered [ 0.254128] acpiphp: Slot [28] registered [ 0.255111] acpiphp: Slot [29] registered [ 0.256107] acpiphp: Slot [30] registered [ 0.257112] acpiphp: Slot [31] registered [ 0.258083] PCI host bridge to bus 0000:00 [ 0.259018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.260022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.261032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.262023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.263023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.264024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.265200] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.267079] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.269694] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.275014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.278040] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.279019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.280021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.281016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.282567] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.283863] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.284049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.285759] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.288015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.297017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.299016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.302732] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.310015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.317017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.331016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.342759] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.356016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.368015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.385014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.396220] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.400015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.406018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.421022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.429451] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.446013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.455013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.472016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.482248] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.488014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.496014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.516000] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.525980] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.537013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.545016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.570017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.583528] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.586378] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.589545] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.591347] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.594204] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.599091] iommu: Default domain type: Passthrough [ 0.601444] SCSI subsystem initialized [ 0.604155] ACPI: bus type USB registered [ 0.606137] usbcore: registered new interface driver usbfs [ 0.608086] usbcore: registered new interface driver hub [ 0.611076] usbcore: registered new device driver usb [ 0.613160] pps_core: LinuxPPS API ver. 1 registered [ 0.616012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.620049] PTP clock support registered [ 0.623102] EDAC MC: Ver: 3.0.0 [ 0.625102] PCI: Using ACPI for IRQ routing [ 0.627549] NetLabel: Initializing [ 0.628000] NetLabel: domain hash size = 128 [ 0.628000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.629076] NetLabel: unlabeled traffic allowed by default [ 0.631061] vgaarb: loaded [ 0.632261] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.634011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.640000] clocksource: Switched to clocksource kvm-clock [ 0.745372] VFS: Disk quotas dquot_6.6.0 [ 0.746943] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.749477] *** VALIDATE ramfs *** [ 0.750541] *** VALIDATE hugetlbfs *** [ 0.752153] pnp: PnP ACPI init [ 0.754881] pnp: PnP ACPI: found 6 devices [ 0.775853] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.779375] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.781818] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.784269] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.786873] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.789519] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.792446] NET: Registered protocol family 2 [ 0.794792] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.799621] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.802656] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.807662] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.810948] TCP: Hash tables configured (established 65536 bind 65536) [ 0.813540] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.816734] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.819659] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.822746] NET: Registered protocol family 1 [ 0.826532] RPC: Registered named UNIX socket transport module. [ 0.828634] RPC: Registered udp transport module. [ 0.830323] RPC: Registered tcp transport module. [ 0.832167] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.835251] NET: Registered protocol family 44 [ 0.836615] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.838168] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.839754] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.841795] PCI: CLS 0 bytes, default 64 [ 0.843081] Unpacking initramfs... [ 2.231716] debug: unmapping init [mem 0xffff9f30bcc54000-0xffff9f30bffbffff] [ 2.235210] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.237970] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.240635] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.760530] Initialise system trusted keyrings [ 2.762723] Key type blacklist registered [ 2.764729] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.773782] zbud: loaded [ 2.777059] *** VALIDATE nfs *** [ 2.778458] *** VALIDATE nfs4 *** [ 2.780483] pstore: using deflate compression [ 2.783756] Platform Keyring initialized [ 2.882729] NET: Registered protocol family 38 [ 2.884805] Key type asymmetric registered [ 2.886630] Asymmetric key parser 'x509' registered [ 2.888781] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.891801] io scheduler mq-deadline registered [ 2.893371] io scheduler kyber registered [ 2.894951] io scheduler bfq registered [ 2.897064] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.900245] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.903257] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.906074] ACPI: Power Button [PWRF] [ 2.911938] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.919388] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.937187] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.948291] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.971970] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.001823] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.030931] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.037206] Non-volatile memory driver v1.3 [ 3.039076] Linux agpgart interface v0.103 [ 3.074328] virtio_blk virtio1: [vda] 146656 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.076592] vda: detected capacity change from 0 to 75087872 [ 3.093545] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.097020] vdb: detected capacity change from 0 to 1073741824 [ 3.113510] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.116758] vdc: detected capacity change from 0 to 2621440000 [ 3.131080] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.134242] vdd: detected capacity change from 0 to 2621440000 [ 3.153200] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.156325] vde: detected capacity change from 0 to 4294967296 [ 3.172524] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.175390] vdf: detected capacity change from 0 to 4294967296 [ 3.182782] libphy: Fixed MDIO Bus: probed [ 3.187236] usbcore: registered new interface driver usbserial_generic [ 3.189315] usbserial: USB Serial support registered for generic [ 3.191291] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.195641] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.197426] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.199928] mousedev: PS/2 mouse device common for all mice [ 3.219235] rtc_cmos 00:05: RTC can wake from S4 [ 3.221934] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.228815] rtc_cmos 00:05: registered as rtc0 [ 3.231341] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.235263] intel_pstate: CPU model not supported [ 3.236348] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.242235] hid: raw HID events driver (C) Jiri Kosina [ 3.243035] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.244294] usbcore: registered new interface driver usbhid [ 3.250406] usbhid: USB HID core driver [ 3.251954] drop_monitor: Initializing network drop monitor service [ 3.254262] Initializing XFRM netlink socket [ 3.256416] NET: Registered protocol family 10 [ 3.259440] Segment Routing with IPv6 [ 3.260614] NET: Registered protocol family 17 [ 3.262078] mpls_gso: MPLS GSO support [ 3.268665] RAS: Correctable Errors collector initialized. [ 3.270633] AVX version of gcm_enc/dec engaged. [ 3.272599] AES CTR mode by8 optimization enabled [ 3.345072] sched_clock: Marking stable (3345028531, 0)->(4355231832, -1010203301) [ 3.349262] registered taskstats version 1 [ 3.352182] Loading compiled-in X.509 certificates [ 3.354441] zswap: loaded using pool lzo/zbud [ 3.379808] Key type big_key registered [ 3.391567] Key type encrypted registered [ 3.393421] ima: No TPM chip found, activating TPM-bypass! [ 3.395751] ima: Allocated hash algorithm: sha1 [ 3.397236] ima: No architecture policies found [ 3.398711] evm: Initialising EVM extended attributes: [ 3.400619] evm: security.selinux [ 3.401901] evm: security.ima [ 3.402918] evm: security.capability [ 3.404333] evm: HMAC attrs: 0x1 [ 3.407501] rtc_cmos 00:05: setting system clock to 2026-08-27 11:13:14 UTC (1787829194) [ 3.414377] debug: unmapping init [mem 0xffffffff9a003000-0xffffffff9a1fffff] [ 3.417622] debug: unmapping init [mem 0xffffffff98d82000-0xffffffff99058fff] [ 3.426081] Write protecting the kernel read-only data: 28672k [ 3.429652] debug: unmapping init [mem 0xffffffff97403000-0xffffffff975fffff] [ 3.432045] debug: unmapping init [mem 0xffffffff97d14000-0xffffffff97dfffff] [ 3.471132] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.481395] systemd[1]: Detected virtualization kvm. [ 3.483646] systemd[1]: Detected architecture x86-64. [ 3.485911] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.510369] systemd[1]: No hostname configured. [ 3.512300] systemd[1]: Set hostname to . [ 3.514513] random: systemd: uninitialized urandom read (16 bytes read) [ 3.517071] systemd[1]: Initializing machine ID from random generator. [ 3.643678] random: systemd: uninitialized urandom read (16 bytes read) [ 3.646499] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.651290] random: systemd: uninitialized urandom read (16 bytes read) [ 3.654159] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.661127] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.244631] device-mapper: uevent: version 1.0.3 [ 4.247087] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.020051] virtio_net virtio0 ens2: renamed from eth0 [ 5.024296] random: fast init done [ 5.070278] scsi host0: ata_piix [ 5.092590] scsi host1: ata_piix [ 5.094465] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.097372] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.267878] dracut-initqueue[591]: RTNETLINK answers: File exists [ 9.916234] random: crng init done [ 9.919694] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.236257] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.348286] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.634723] SELinux: Disabled at runtime. [ 11.699096] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.708597] systemd[1]: Detected virtualization kvm. [ 11.710486] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.182269] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.185438] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.189051] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.193048] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.197209] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.203817] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.209833] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages Fil[ 12.315619] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS e System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.607218] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.876397] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.881392] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.032930] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.050277] EDAC sbridge: Ver: 1.1.2 [ 14.834431] Key type dns_resolver registered [ 15.143303] NFS: Registering the id_resolver key type [ 15.145411] Key type id_resolver registered [ 15.146978] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg617-server login: [ 45.505050] hrtimer: interrupt took 7562679 ns [ 68.124533] libcfs: loading out-of-tree module taints kernel. [ 68.170440] Key type ._llcrypt registered [ 68.176061] Key type .llcrypt registered [ 68.289642] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 85.430280] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 86.729591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 86.743523] alg: No test for adler32 (adler32-zlib) [ 88.184576] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 89.039163] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 90.887218] Key type lgssc registered [ 93.079987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 106.688813] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 149.151665] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 163.638768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 163.682741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.982175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 165.022493] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 165.148916] Lustre: lustre-MDT0000: new disk, initializing [ 165.248939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 165.272692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 170.281929] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.356131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 183.442941] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 183.473173] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 183.476891] Lustre: Skipped 1 previous similar message [ 183.577407] Lustre: lustre-MDT0001: new disk, initializing [ 183.666168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 183.698134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 183.714262] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 187.934319] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.770842] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 202.783695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 203.059407] Lustre: lustre-OST0000: new disk, initializing [ 203.064076] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 203.072290] Lustre: 8452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 203.154102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 206.523991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 206.534161] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 206.648691] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 210.147106] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.600361] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 225.786670] Lustre: lustre-OST0001: new disk, initializing [ 225.794614] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 225.806965] Lustre: 9525:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 225.928772] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 231.489039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 231.499817] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 231.575600] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 233.446703] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.220711] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 254.467429] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 261.473051] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing check_logdir /tmp/testlogs/ [ 265.783838] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing yml_node [ 270.990969] Lustre: DEBUG MARKER: Client: 2.17.57.64 [ 273.696715] Lustre: DEBUG MARKER: MDS: 2.17.57.64 [ 276.679626] Lustre: DEBUG MARKER: OSS: 2.17.57.64 [ 278.530683] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 07:17:48 EDT 2026 [ 298.818387] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 300.315367] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 318.433765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.442694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 318.472517] Lustre: Skipped 1 previous similar message [ 318.505719] Lustre: Skipped 3 previous similar messages [ 323.552550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 323.566422] Lustre: Skipped 3 previous similar messages [ 324.478358] Lustre: server umount lustre-MDT0000 complete [ 328.499726] LustreError: 7460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829519 with bad export cookie 6520582152351695597 [ 328.500861] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 328.508440] LustreError: 7460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 328.672230] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 328.676949] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 328.698632] Lustre: Skipped 3 previous similar messages [ 328.721879] Lustre: Skipped 1 previous similar message [ 333.810738] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.814685] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 333.853471] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 333.870093] Lustre: Skipped 1 previous similar message [ 334.982079] Lustre: server umount lustre-MDT0001 complete [ 339.185848] Lustre: server umount lustre-OST0000 complete [ 343.608990] Lustre: server umount lustre-OST0001 complete [ 351.566125] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 362.386451] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 398.709054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 398.961311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 398.999645] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 399.064898] Lustre: lustre-MDT0000: new disk, initializing [ 399.128252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 399.146624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 403.871609] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 415.271156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.423218] Lustre: 15075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 415.477873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 415.481822] Lustre: Skipped 1 previous similar message [ 415.560323] Lustre: lustre-MDT0001: new disk, initializing [ 415.731798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 415.769520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 415.779620] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 421.662375] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 432.705735] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 439.606171] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 447.790657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 448.186241] Lustre: lustre-OST0000: new disk, initializing [ 448.190788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 448.204644] Lustre: 17044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 448.322063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.590503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 449.601702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 449.736059] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 455.391083] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 466.648610] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 472.781905] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 473.012882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 478.576350] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 478.860355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 485.865526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 485.875360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.881646] LustreError: Skipped 1 previous similar message [ 485.882470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 485.917205] Lustre: Skipped 1 previous similar message [ 487.340520] Lustre: server umount lustre-OST0000 complete [ 492.514203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 492.524721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.104674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 496.111680] Lustre: Skipped 2 previous similar messages [ 498.171154] Lustre: server umount lustre-MDT0000 complete [ 501.221192] LustreError: 15088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.245300] LustreError: 15088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 502.056625] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829693 with bad export cookie 6520582152351697130 [ 502.073294] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 502.077140] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 502.392220] Lustre: server umount lustre-MDT0001 complete [ 506.889732] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 07:21:37 (1787829697) [ 509.245695] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 511.048608] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 07:21:41 (1787829701) [ 518.346995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 518.993540] LustreError: 19454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.057420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 523.548210] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 524.257516] LustreError: 19454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.288499] LustreError: 19454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 529.378596] LustreError: 19455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.282702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 537.563964] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 546.929487] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 552.094136] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 558.968619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 565.511625] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 575.024788] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 589.280827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 589.285298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.306497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 589.316809] Lustre: Skipped 2 previous similar messages [ 589.329973] Lustre: Skipped 2 previous similar messages [ 593.518683] Lustre: server umount lustre-OST0000 complete [ 599.526605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 599.546109] Lustre: Skipped 1 previous similar message [ 612.319245] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 612.562965] Lustre: server umount lustre-MDT0000 complete [ 614.887048] LustreError: 19451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 614.900802] LustreError: 19451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 616.274988] LustreError: 19437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829807 with bad export cookie 6520582152351698012 [ 616.288319] LustreError: 19437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 616.293798] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 616.584260] Lustre: server umount lustre-MDT0001 complete [ 628.605639] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 631.998376] Key type lgssc unregistered [ 632.312848] LNet: 23007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.319948] LNetError: 23007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 633.381864] LNet: Removed LNI 192.168.206.117@tcp [ 634.545245] Key type .llcrypt unregistered [ 634.552462] Key type ._llcrypt unregistered [ 662.186773] Key type ._llcrypt registered [ 662.188875] Key type .llcrypt registered [ 662.299830] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 663.196263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 663.215631] alg: No test for adler32 (adler32-zlib) [ 664.253814] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 664.668575] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 666.439250] Key type lgssc registered [ 667.522273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 675.569672] loop: module loaded [ 675.652447] loop0: detected capacity change from 0 to 204800000 [ 675.950309] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 675.974923] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 676.231470] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 682.460993] loop0: detected capacity change from 0 to 204800000 [ 682.690434] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 682.704443] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 682.721452] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 687.832134] loop0: detected capacity change from 0 to 204800000 [ 693.519578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 693.565922] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 694.877275] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 694.972164] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 695.065671] Lustre: lustre-MDT0000: new disk, initializing [ 695.138846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 695.174775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 699.877922] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 708.186530] loop1: detected capacity change from 0 to 204800000 [ 713.769332] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 714.107952] Lustre: lustre-OST07c6: new disk, initializing [ 714.113919] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 714.122139] Lustre: Skipped 1 previous similar message [ 714.127276] Lustre: 26197:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 714.214120] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 719.445973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 719.456467] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 719.491100] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 720.020580] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 724.419345] Lustre: 25184:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 724.425505] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 724.430359] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 724.436266] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 724.443744] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 724.452204] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 739.777190] Lustre: 25184:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 739.792224] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 739.802042] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 739.811160] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 739.819722] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 739.827946] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 744.589728] Lustre: 26491:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 744.600608] Lustre: 26491:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 744.608740] Lustre: 26491:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 744.615964] Lustre: 26491:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.621598] Lustre: 26491:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 744.628637] Lustre: 26491:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.646569] Lustre: 26491:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 744.656317] Lustre: 26491:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.668159] Lustre: 26491:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 744.676378] Lustre: 26491:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.685923] Lustre: 26491:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 744.690462] Lustre: 26491:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.904352] Lustre: 25185:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 747.920705] Lustre: 25185:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 747.927847] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 747.933820] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.939881] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 747.945978] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.950394] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 747.957912] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.967319] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 747.972318] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.976383] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 747.979035] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.298337] Lustre: 25186:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 752.307378] Lustre: 25186:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 752.315103] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 752.322290] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.336464] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 752.342035] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.348487] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 752.356618] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.363963] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 752.370958] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.381950] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 752.387597] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.984825] Lustre: 25184:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 763.002673] Lustre: 25184:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 763.018991] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 763.029839] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 763.041709] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 763.050687] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 763.058595] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 763.069794] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 763.084608] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 763.098109] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 763.113175] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 763.124452] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 811.490296] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 811.517728] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 815.611148] Lustre: server umount lustre-OST07c6 complete [ 826.205848] Lustre: server umount lustre-MDT0000 complete [ 842.327910] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 844.968696] Key type lgssc unregistered [ 845.269496] LNet: 28756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 845.280824] LNetError: 28756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 845.309378] LNet: Removed LNI 192.168.206.117@tcp [ 846.226388] Key type .llcrypt unregistered [ 846.231688] Key type ._llcrypt unregistered [ 863.039848] Key type ._llcrypt registered [ 863.043208] Key type .llcrypt registered [ 863.206291] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 864.491336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 864.527973] alg: No test for adler32 (adler32-zlib) [ 865.561325] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 865.763757] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 867.433399] Key type lgssc registered [ 868.503718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 876.385684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 876.430112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.980675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 882.062112] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 888.287889] LustreError: 30020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 888.298165] LustreError: 30020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 890.404128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 890.841455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 895.487869] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 903.185887] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 909.077650] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 915.774892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 916.145530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 922.325826] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 923.343203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 930.484128] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 951.269451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 951.272516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 951.281527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 951.287084] Lustre: Skipped 1 previous similar message [ 952.211984] Lustre: server umount lustre-OST0000 complete [ 956.385136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 956.387372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 956.403996] Lustre: Skipped 1 previous similar message [ 956.414914] Lustre: Skipped 1 previous similar message [ 961.522353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 961.525229] Lustre: Skipped 1 previous similar message [ 962.604154] Lustre: server umount lustre-MDT0000 complete [ 966.629336] LustreError: 30014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 966.655590] LustreError: 30014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 966.685443] LustreError: 30000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830157 with bad export cookie 3609526505323710904 [ 966.693754] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 966.710516] LustreError: 30000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 967.363958] Lustre: server umount lustre-MDT0001 complete [ 979.618920] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 982.620411] Key type lgssc unregistered [ 982.890077] LNet: 33759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.905193] LNetError: 33759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.920774] LNet: Removed LNI 192.168.206.117@tcp [ 983.726712] Key type .llcrypt unregistered [ 983.728796] Key type ._llcrypt unregistered [ 1003.361938] Key type ._llcrypt registered [ 1003.364521] Key type .llcrypt registered [ 1003.458565] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 1004.640261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1004.696798] alg: No test for adler32 (adler32-zlib) [ 1006.013824] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 1006.397151] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 1008.183245] Key type lgssc registered [ 1010.015769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1020.063577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1020.109920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1021.649830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1026.362987] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1031.143110] LustreError: 35031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1031.162489] LustreError: 35031:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1035.203285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1035.707240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1040.423591] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1048.887209] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1054.864121] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1061.857509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1062.303583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1066.430438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1068.238413] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1076.337715] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1091.041437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1091.064563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1091.069916] Lustre: Skipped 1 previous similar message [ 1096.168578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1096.184357] Lustre: Skipped 1 previous similar message [ 1097.192036] Lustre: server umount lustre-OST0000 complete [ 1102.304609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1102.310306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1102.318110] Lustre: Skipped 1 previous similar message [ 1102.321488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1106.401136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1106.411296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1106.424958] Lustre: Skipped 1 previous similar message [ 1107.848634] Lustre: server umount lustre-MDT0000 complete [ 1111.524357] LustreError: 35031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1111.561822] LustreError: 35031:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1111.874235] LustreError: 35010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830302 with bad export cookie 7427645617886240650 [ 1111.879255] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1111.885752] LustreError: 35010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1112.237363] Lustre: server umount lustre-MDT0001 complete [ 1124.787996] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 1127.681621] Key type lgssc unregistered [ 1128.001726] LNet: 38593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1128.010636] LNetError: 38593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1128.025776] LNet: Removed LNI 192.168.206.117@tcp [ 1128.935215] Key type .llcrypt unregistered [ 1128.937116] Key type ._llcrypt unregistered [ 1153.398538] Key type ._llcrypt registered [ 1153.403971] Key type .llcrypt registered [ 1153.567437] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 1154.425278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1154.538029] alg: No test for adler32 (adler32-zlib) [ 1155.569616] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 1155.849461] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 1157.583181] Key type lgssc registered [ 1158.800605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1166.420496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1166.434150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1168.131357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1173.126289] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1178.617061] LustreError: 40016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1178.628157] LustreError: 40016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1182.089775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1182.405081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1186.671513] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1194.581192] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1200.944775] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1208.981346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1209.193602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1213.296237] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1215.356194] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1225.191848] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1237.219770] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.17@tcp (stopping) [ 1239.024446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1239.026937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1239.049733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1239.057488] Lustre: Skipped 1 previous similar message [ 1240.904316] Lustre: server umount lustre-MDT0000 complete [ 1242.312285] LustreError: 40013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1244.133499] LustreError: 40017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1244.151946] LustreError: 40017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1245.296123] LustreError: 39998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830436 with bad export cookie 6455031191176994132 [ 1245.308876] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1245.316280] LustreError: 39998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1247.443613] LustreError: 40011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1247.488201] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.17@tcp (stopping) [ 1247.517026] Lustre: Skipped 2 previous similar messages [ 1249.283442] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1249.302968] Lustre: Skipped 1 previous similar message [ 1250.273914] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1250.277107] Lustre: Skipped 1 previous similar message [ 1251.776448] Lustre: server umount lustre-MDT0001 complete [ 1268.715376] Lustre: server umount lustre-OST0000 complete [ 1287.693731] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 1289.937382] Key type lgssc unregistered [ 1290.193445] LNet: 43880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1290.202794] LNetError: 43880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1290.225341] LNet: Removed LNI 192.168.206.117@tcp [ 1291.003503] Key type .llcrypt unregistered [ 1291.005604] Key type ._llcrypt unregistered [ 1310.985784] Key type ._llcrypt registered [ 1310.987662] Key type .llcrypt registered [ 1311.082959] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 1312.218912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1312.349099] alg: No test for adler32 (adler32-zlib) [ 1313.450980] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 1313.708564] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 1315.383239] Key type lgssc registered [ 1316.437127] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1324.167196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1324.195978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1325.780688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1330.464877] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1335.266917] LustreError: 45288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1335.299065] LustreError: 45288:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1339.199308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1339.549881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1344.187590] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1352.888434] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1358.819907] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1365.717136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1366.105899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1370.237673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1372.080772] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1379.415801] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1390.048840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1390.051874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1390.061239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1390.069280] Lustre: Skipped 1 previous similar message [ 1390.081132] Lustre: Skipped 1 previous similar message [ 1391.245568] Lustre: server umount lustre-OST0000 complete [ 1410.529876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1410.542036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1414.757878] Lustre: server umount lustre-MDT0000 complete [ 1415.655346] LustreError: 46017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1415.672232] LustreError: 46017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1418.657260] LustreError: 45270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830609 with bad export cookie 10790159204779121436 [ 1418.669344] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1418.670390] LustreError: 45270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1425.044328] Lustre: server umount lustre-MDT0001 complete [ 1434.745529] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 1437.492171] Key type lgssc unregistered [ 1437.780547] LNet: 48998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1437.787458] LNetError: 48998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1437.800807] LNet: Removed LNI 192.168.206.117@tcp [ 1438.506262] Key type .llcrypt unregistered [ 1438.508560] Key type ._llcrypt unregistered [ 1462.452811] Key type ._llcrypt registered [ 1462.456693] Key type .llcrypt registered [ 1462.540362] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 1463.656286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1463.669448] alg: No test for adler32 (adler32-zlib) [ 1464.782962] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 1465.133088] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 1466.839169] Key type lgssc registered [ 1468.143387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1475.621193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1475.665816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1477.041832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1481.133632] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1487.333882] LustreError: 50414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1487.352586] LustreError: 50414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1489.658306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1490.160197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1495.008442] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1503.236537] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1509.155434] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1515.502335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1515.841973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1521.328570] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1521.973672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1530.357142] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1539.070383] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1543.472290] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1556.054192] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-27 7h39m05s [ 1558.163908] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1561.056791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1561.079446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1561.089408] Lustre: Skipped 2 previous similar messages [ 1561.110907] Lustre: Skipped 2 previous similar messages [ 1562.149296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1562.168089] Lustre: Skipped 2 previous similar messages [ 1563.909930] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.17@tcp (stopping) [ 1563.920504] Lustre: Skipped 1 previous similar message [ 1566.530900] Lustre: server umount lustre-MDT0000 complete [ 1568.225872] LustreError: 50415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1568.248143] LustreError: 50415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1569.994492] LustreError: 50409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1570.158726] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1574.372906] LustreError: 50414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1574.389221] LustreError: 50414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1577.857040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1577.993669] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1578.296684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1578.366847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1579.324726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1583.179475] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1586.159259] Lustre: DEBUG MARKER: Wait for df (36677) ... [ 1587.430431] Lustre: DEBUG MARKER: done [ 1593.824851] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1593.841604] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1593.881953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1594.853315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1598.905403] Lustre: server umount lustre-OST0000 complete [ 1605.090207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1605.091721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1605.091814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1605.098906] Lustre: Skipped 1 previous similar message [ 1605.101812] Lustre: Skipped 2 previous similar messages [ 1608.920119] Lustre: server umount lustre-MDT0000 complete [ 1612.259340] LustreError: 50415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1612.274386] LustreError: 50415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1612.367361] LustreError: 50394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830803 with bad export cookie 14623936295115102546 [ 1612.387047] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1612.709581] Lustre: server umount lustre-MDT0001 complete [ 1625.141690] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 1628.203936] Key type lgssc unregistered [ 1628.711815] LNet: 55342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1628.728509] LNetError: 55342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1628.742066] LNet: Removed LNI 192.168.206.117@tcp [ 1629.560326] Key type .llcrypt unregistered [ 1629.562965] Key type ._llcrypt unregistered [ 1667.064632] Key type ._llcrypt registered [ 1667.067101] Key type .llcrypt registered [ 1667.200513] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 1668.291131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1668.322953] alg: No test for adler32 (adler32-zlib) [ 1669.350838] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 1669.606838] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 1671.223177] Key type lgssc registered [ 1672.295036] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1679.415715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1679.447234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1680.721288] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1680.772542] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1680.976962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1684.308341] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1692.336935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1692.430590] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1692.463782] Lustre: 57529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1692.744251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1697.927502] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1705.387768] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1711.935125] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1718.808667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1718.975180] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1719.238468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1725.158117] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1726.116368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1732.951956] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1738.851456] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-27 7h42m09s [ 1740.324906] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1746.928357] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1749.291484] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1749.336946] LustreError: 59688:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1769.736024] Lustre: lustre-MDT0000: Client fcd684cc-e3a4-4445-8e93-f60282bfbf25 (at 192.168.206.17@tcp) reconnecting [ 1789.359186] LustreError: 59688:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1791.866897] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1794.800217] Lustre: DEBUG MARKER: done [ 1802.730497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1802.742161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1802.757814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1803.233134] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1803.250636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1803.280527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1803.285534] Lustre: Skipped 1 previous similar message [ 1808.378483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1808.384567] Lustre: Skipped 1 previous similar message [ 1808.877880] Lustre: server umount lustre-OST0000 complete [ 1815.520156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1815.539506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1815.557442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1819.683058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1819.694104] Lustre: Skipped 4 previous similar messages [ 1819.954911] Lustre: server umount lustre-MDT0000 complete [ 1824.658945] LustreError: 57522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831015 with bad export cookie 11458925162367513758 [ 1824.664468] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1824.686332] LustreError: 57522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1825.093745] Lustre: server umount lustre-MDT0001 complete [ 1837.980319] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 1840.783373] Key type lgssc unregistered [ 1841.011319] LNet: 61864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.019782] LNetError: 61864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.034792] LNet: Removed LNI 192.168.206.117@tcp [ 1841.973089] Key type .llcrypt unregistered [ 1841.975347] Key type ._llcrypt unregistered [ 1886.869835] Key type ._llcrypt registered [ 1886.871809] Key type .llcrypt registered [ 1886.958628] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 1888.044952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1888.074955] alg: No test for adler32 (adler32-zlib) [ 1889.226115] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 1889.505834] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 1891.183868] Key type lgssc registered [ 1892.300265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1895.384437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1895.425790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1896.695585] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1896.719749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1896.928538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1897.820159] Lustre: Failing over lustre-MDT0000 [ 1898.125728] Lustre: server umount lustre-MDT0000 complete [ 1904.728708] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 07:44:54 (1787831094) [ 1911.934451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1912.292988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1917.940415] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1926.675896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1926.868034] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1926.906366] Lustre: 64664:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1927.215683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1932.484318] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1939.949721] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1945.941977] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1952.367414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1952.516090] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1952.741359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1958.853815] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1959.140903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1965.871885] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1972.169499] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1979.878680] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1979.894178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1979.900621] Lustre: Skipped 1 previous similar message [ 1979.908125] Lustre: Skipped 2 previous similar messages [ 1985.014314] Lustre: server umount lustre-MDT0000 complete [ 1988.625546] LustreError: 64657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831179 with bad export cookie 4894595113527350474 [ 1988.639786] LustreError: 64657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1988.639969] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1989.153268] Lustre: server umount lustre-MDT0001 complete [ 1992.774989] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1997.611474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2005.699080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2006.496027] Lustre: 63591:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787831181/real 1787831181] req@0000000038b4d45e x1874676762582656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787831197 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2006.526465] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2006.544424] Lustre: Skipped 1 previous similar message [ 2009.567247] Lustre: 63590:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787831184/real 1787831184] req@00000000cf615e3e x1874676762582784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787831200 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2011.615246] Lustre: 63589:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787831186/real 1787831186] req@000000004820fa25 x1874676762582912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787831202 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2013.731783] LustreError: 63587:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000069ba7b23 x1874676762583680/t0(0) o250->MGC192.168.206.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2014.104684] LustreError: 66516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2014.119241] LustreError: 66516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2014.227238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2014.359994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2015.267632] Lustre: 63589:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787831189/real 1787831189] req@000000004fd56f78 x1874676762583168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787831205 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2015.271022] LustreError: 66517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.298557] LustreError: 68435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.353875] LustreError: 68435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2018.985345] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2020.320692] Lustre: 63590:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787831194/real 1787831194] req@00000000784caef6 x1874676762583296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787831210 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2021.345864] LustreError: 68436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2021.367854] LustreError: 68436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2026.465585] LustreError: 68832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2026.494479] LustreError: 68832:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2026.761681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2027.157674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2031.284306] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2032.617091] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2035.694093] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2037.739545] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2042.282193] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2051.041564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2051.044990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2051.058263] Lustre: Skipped 2 previous similar messages [ 2051.079968] Lustre: Skipped 2 previous similar messages [ 2054.851593] Lustre: server umount lustre-MDT0000 complete [ 2056.171192] LustreError: 68832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2056.198186] LustreError: 68832:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2058.191516] LustreError: 64658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831249 with bad export cookie 4894595113527352749 [ 2058.196428] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2058.219702] LustreError: 64658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2058.562878] Lustre: server umount lustre-MDT0001 complete [ 2064.938483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2075.822025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.663692] Lustre: 63589:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787831252/real 1787831252] req@0000000088d5a0d6 x1874676762610304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787831268 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2077.700520] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2083.881432] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x43ed1852b5292310 [ 2083.892494] Lustre: MGC192.168.206.117@tcp: Connection restored to 0@lo (at 0@lo) [ 2084.098235] LustreError: 66517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2084.186553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2084.252071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2088.435434] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2096.077511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2100.985528] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2101.743632] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2106.922198] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2107.986806] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2113.351159] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2119.219225] Lustre: server umount lustre-MDT0000 complete [ 2122.215202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2122.215939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2122.220513] LustreError: 72362:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2122.220526] LustreError: 72362:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2122.225138] Lustre: Skipped 1 previous similar message [ 2122.632625] LustreError: 64658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831313 with bad export cookie 4894595113527354128 [ 2122.637382] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2122.638238] LustreError: 64658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2124.900858] Lustre: server umount lustre-MDT0001 complete [ 2131.302853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2134.675924] Lustre: DEBUG MARKER: files compared the same [ 2146.774049] Lustre: server umount lustre-OST0000 complete [ 2165.774261] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 2168.959574] Key type lgssc unregistered [ 2169.228733] LNet: 74394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2169.233860] LNetError: 74394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2169.248848] LNet: Removed LNI 192.168.206.117@tcp [ 2169.976346] Key type .llcrypt unregistered [ 2169.981389] Key type ._llcrypt unregistered [ 2195.710268] Key type ._llcrypt registered [ 2195.715261] Key type .llcrypt registered [ 2195.845248] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 2196.815133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2196.824464] alg: No test for adler32 (adler32-zlib) [ 2197.845854] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 2198.132410] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 2199.802082] Key type lgssc registered [ 2201.225153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2208.540994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2208.575885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2209.963403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2213.907191] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug +malloc all [ 2220.518757] LustreError: 75814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2220.540920] LustreError: 75814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2221.807511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2222.210903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2226.634609] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug +malloc all [ 2233.718218] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2238.418926] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2244.105490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2244.346112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2250.293193] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug +malloc all [ 2255.589192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2256.070459] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2266.081093] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2266.083759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2266.095904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2266.109556] Lustre: Skipped 1 previous similar message [ 2266.120311] Lustre: Skipped 1 previous similar message [ 2269.492125] Lustre: server umount lustre-OST0000 complete [ 2273.248186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2273.261130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2273.274938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2276.320805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2276.322468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2276.362528] Lustre: Skipped 1 previous similar message [ 2281.440803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2281.445226] Lustre: Skipped 1 previous similar message [ 2286.560818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2286.567183] Lustre: Skipped 1 previous similar message [ 2287.071258] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2287.316239] Lustre: server umount lustre-MDT0000 complete [ 2290.635485] LustreError: 75794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831481 with bad export cookie 7342134148407747238 [ 2290.652923] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2290.948801] Lustre: server umount lustre-MDT0001 complete [ 2302.570733] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 2305.123405] Key type lgssc unregistered [ 2305.370231] LNet: 79350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2305.379851] LNetError: 79350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2305.401786] LNet: Removed LNI 192.168.206.117@tcp [ 2306.229712] Key type .llcrypt unregistered [ 2306.234508] Key type ._llcrypt unregistered [ 2327.823833] Key type ._llcrypt registered [ 2327.830179] Key type .llcrypt registered [ 2328.006264] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 2329.323946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2329.376467] alg: No test for adler32 (adler32-zlib) [ 2330.447603] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 2330.709539] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 2332.375230] Key type lgssc registered [ 2333.410506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2340.395251] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2340.410397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2355.936433] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2361.055514] LustreError: 80741:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.117@tcp: failed processing log, type 4: rc = -110 [ 2387.679171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2393.992419] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2400.964623] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2409.131099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2409.328402] Lustre: *** cfs_fail_loc=706, val=0*** [ 2409.753963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2409.866576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2414.566562] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2415.073050] LustreError: 80767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.121425] LustreError: 80767:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2420.197149] LustreError: 80767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2422.248778] LustreError: 81826:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2422.262187] LustreError: 81826:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2423.338067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2423.709695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2428.333740] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2436.348508] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2442.635598] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2445.804441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2445.815256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2445.833710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2449.392471] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2449.394148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2449.417864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2450.875156] Lustre: server umount lustre-OST0000 complete [ 2454.500744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2454.501992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2454.502181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2454.513094] Lustre: Skipped 1 previous similar message [ 2454.515907] Lustre: Skipped 1 previous similar message [ 2459.620930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2459.630399] Lustre: Skipped 2 previous similar messages [ 2460.691364] Lustre: server umount lustre-MDT0000 complete [ 2463.479968] LustreError: 80748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831654 with bad export cookie 8995124421605084424 [ 2463.488171] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2463.495215] LustreError: 80748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2463.750320] Lustre: server umount lustre-MDT0001 complete [ 2471.301805] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 2473.631660] Key type lgssc unregistered [ 2473.862857] LNet: 84357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2473.875569] LNetError: 84357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2474.917906] LNet: Removed LNI 192.168.206.117@tcp [ 2475.818950] Key type .llcrypt unregistered [ 2475.825656] Key type ._llcrypt unregistered [ 2497.511813] Key type ._llcrypt registered [ 2497.515219] Key type .llcrypt registered [ 2497.649396] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 2498.647224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2498.733481] alg: No test for adler32 (adler32-zlib) [ 2499.800166] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 2500.077261] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 2501.759193] Key type lgssc registered [ 2502.822504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2509.551683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2509.572319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2514.484472] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2520.764093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2521.183591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2524.749890] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2526.178617] LustreError: 86459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2526.190051] LustreError: 86459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2531.297631] LustreError: 86462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2531.320437] LustreError: 86462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2532.595563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2532.895434] LustreError: 86457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2532.912240] LustreError: 86457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2532.927526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.958908] LustreError: 86459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2537.974976] LustreError: 86459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2538.611347] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2543.075885] LustreError: 87208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.088199] LustreError: 87208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2545.594117] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2545.607090] Lustre: MGS: Client c53236db-d6d4-402a-999b-9f28319c1f29 (at 0@lo) reconnecting [ 2545.626668] LustreError: 86473:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2545.793263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2545.855838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2549.878599] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2562.539871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2562.557279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2562.576086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2562.584264] Lustre: Skipped 1 previous similar message [ 2567.656731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2567.663277] Lustre: Skipped 1 previous similar message [ 2568.403798] Lustre: server umount lustre-OST0000 complete [ 2572.776666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2572.783953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2572.784096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2572.793356] Lustre: Skipped 2 previous similar messages [ 2572.807929] Lustre: Skipped 1 previous similar message [ 2577.889308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2577.898165] Lustre: Skipped 1 previous similar message [ 2578.162340] Lustre: server umount lustre-MDT0000 complete [ 2582.315810] Lustre: server umount lustre-MDT0001 complete [ 2586.211742] Lustre: server umount MGS complete [ 2597.643707] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 07:56:27 (1787831787) [ 2624.219410] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 2633.306792] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 2672.953616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2677.890106] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2687.378620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2687.606667] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2687.706114] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2687.789552] Lustre: lustre-MDT0001: new disk, initializing [ 2687.855375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2687.885948] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2687.898124] Lustre: 93633:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2687.902407] LustreError: 93633:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2691.854796] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2702.571074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2702.814960] Lustre: lustre-OST0000: new disk, initializing [ 2702.819565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2702.826597] Lustre: 94401:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2702.916170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2709.013363] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2711.072392] Lustre: 94823:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000088d5a0d6 x1874677403261568/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787831918 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2718.652360] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2725.765855] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2725.781518] Lustre: MGS: Client af1a39c1-50ec-46b1-95e6-51c87152d9e6 (at 0@lo) reconnecting [ 2725.855169] Lustre: 95446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2725.869403] LustreError: 93619:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2725.898883] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2725.961618] Lustre: lustre-MDT0000: new disk, initializing [ 2726.142294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2726.215576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2726.247145] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2727.337278] Lustre: 95470:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000052c19d9f x1874677403268608/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787831934 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2731.166377] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2732.449618] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2736.517616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2736.539957] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2736.549131] Lustre: Skipped 1 previous similar message [ 2736.569387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2762.213174] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2762.224235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.231781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2765.754812] Lustre: server umount lustre-OST0000 complete [ 2768.869473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2768.885476] Lustre: Skipped 2 previous similar messages [ 2773.984845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2773.997426] Lustre: Skipped 5 previous similar messages [ 2774.674345] Lustre: server umount lustre-MDT0000 complete [ 2784.439205] Lustre: server umount lustre-MDT0001 complete [ 2787.701699] Lustre: server umount MGS complete [ 2797.497970] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 07:59:47 (1787831987) [ 2814.693779] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 2817.586804] Key type lgssc unregistered [ 2817.916932] LNet: 97970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2817.924122] LNetError: 97970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2817.939971] LNet: Removed LNI 192.168.206.117@tcp [ 2818.861864] Key type .llcrypt unregistered [ 2818.867614] Key type ._llcrypt unregistered [ 2834.586988] Key type ._llcrypt registered [ 2834.590179] Key type .llcrypt registered [ 2834.678559] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 2835.795750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2835.835503] alg: No test for adler32 (adler32-zlib) [ 2836.903220] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 2837.150427] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 2838.847169] Key type lgssc registered [ 2840.127584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2847.603151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2847.620965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2848.984181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2853.082352] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2859.493344] LustreError: 99186:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2859.511328] LustreError: 99186:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2861.005111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.278737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2865.180828] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2871.729511] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2876.830929] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2882.861859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2883.193181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2888.681595] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2894.264949] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2894.452031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2901.984606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2901.992930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2902.028495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2905.057553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2905.065909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2905.074707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2910.176891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2910.185775] Lustre: Skipped 2 previous similar messages [ 2915.295211] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2915.321047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2915.331035] Lustre: Skipped 1 previous similar message [ 2915.550589] Lustre: server umount lustre-OST0000 complete [ 2920.418116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2920.428027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2920.447668] Lustre: Skipped 1 previous similar message [ 2920.459068] Lustre: Skipped 1 previous similar message [ 2925.686565] Lustre: server umount lustre-MDT0000 complete [ 2929.233774] LustreError: 99920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832120 with bad export cookie 6307329631844506585 [ 2929.251676] LustreError: 99920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2929.257284] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2929.625348] Lustre: server umount lustre-MDT0001 complete [ 2939.925628] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 2942.568427] Key type lgssc unregistered [ 2942.890521] LNet: 102737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2942.905294] LNetError: 102737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2942.926582] LNet: Removed LNI 192.168.206.117@tcp [ 2943.801321] Key type .llcrypt unregistered [ 2943.806381] Key type ._llcrypt unregistered [ 2958.082517] Key type ._llcrypt registered [ 2958.086707] Key type .llcrypt registered [ 2958.225333] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 2959.237546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2959.390771] alg: No test for adler32 (adler32-zlib) [ 2960.443209] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 2960.626983] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 2962.319758] Key type lgssc registered [ 2963.642987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2970.190106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2970.200330] LustreError: 103798:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2970.227335] LustreError: 103801:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2970.235108] LustreError: 103798:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4983 [ 2970.241933] LustreError: 103798:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2970.265068] LustreError: 103798:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2970.265679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2971.852615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2979.361127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2979.621884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2983.958356] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2989.467227] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing lsmod [ 2991.698300] LustreError: 105005:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2991.726220] LustreError: 105008:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2991.735968] LustreError: 105005:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4969 [ 2991.747309] LustreError: 105005:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2991.757377] LustreError: 105005:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2991.758669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2992.111394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2995.168494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2995.191220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2995.213143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2997.219188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2997.236673] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2997.261398] Lustre: Skipped 1 previous similar message [ 3000.912035] Lustre: server umount lustre-MDT0000 complete [ 3002.336669] LustreError: 103832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3002.364350] LustreError: 103832:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3004.163269] LustreError: 103812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832195 with bad export cookie 8709879394299767194 [ 3004.172609] LustreError: 103812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3004.172795] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3004.482893] Lustre: server umount lustre-MDT0001 complete [ 3017.350928] Lustre: server umount lustre-OST0000 complete [ 3024.988311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3025.279690] LustreError: 106071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3025.364146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3029.980114] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3030.499903] LustreError: 106072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3035.619535] LustreError: 106071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3038.076798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3038.494503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3042.077214] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3048.195596] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3052.667808] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3059.175201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3059.554969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3065.125152] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3070.850871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3073.274872] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3080.160724] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3080.172532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3080.197265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3080.209258] Lustre: Skipped 2 previous similar messages [ 3085.873719] Lustre: server umount lustre-OST0000 complete [ 3089.890521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3089.902932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3089.921262] Lustre: Skipped 1 previous similar message [ 3089.929724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3089.934124] Lustre: Skipped 2 previous similar messages [ 3096.546123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3096.552923] Lustre: Skipped 3 previous similar messages [ 3103.199689] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3103.398531] Lustre: server umount lustre-MDT0000 complete [ 3106.458800] LustreError: 106051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832297 with bad export cookie 8709879394299767901 [ 3106.468985] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3106.755580] Lustre: server umount lustre-MDT0001 complete [ 3118.173822] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 3120.707053] Key type lgssc unregistered [ 3120.985771] LNet: 109608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3120.991027] LNetError: 109608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3121.001216] LNet: Removed LNI 192.168.206.117@tcp [ 3121.912905] Key type .llcrypt unregistered [ 3121.918122] Key type ._llcrypt unregistered [ 3148.628946] Key type ._llcrypt registered [ 3148.631877] Key type .llcrypt registered [ 3148.809649] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 3150.739070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3150.840612] alg: No test for adler32 (adler32-zlib) [ 3152.175817] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 3152.569811] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 3154.239797] Key type lgssc registered [ 3155.349458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3163.905877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3163.958611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3165.546706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3170.004585] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3175.907083] LustreError: 111027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3175.928367] LustreError: 111027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3176.845699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3177.136317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3181.578128] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3190.927478] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3199.085745] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3205.354642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3205.625882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3211.786624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3212.001551] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3219.422780] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3225.754844] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3230.279603] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3237.347118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3237.348521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3237.354212] LustreError: Skipped 1 previous similar message [ 3237.356301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3237.363533] Lustre: Skipped 1 previous similar message [ 3237.377091] Lustre: Skipped 1 previous similar message [ 3240.879191] Lustre: server umount lustre-OST0000 complete [ 3247.586313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3247.589250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3247.618749] Lustre: Skipped 1 previous similar message [ 3247.638584] Lustre: Skipped 1 previous similar message [ 3251.057588] Lustre: server umount lustre-MDT0000 complete [ 3252.709155] LustreError: 112040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3252.724169] LustreError: 112040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3255.259492] LustreError: 111008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832446 with bad export cookie 14432750311453881822 [ 3255.271851] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3255.281753] LustreError: 111008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3255.832793] Lustre: server umount lustre-MDT0001 complete [ 3266.692893] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 3269.079974] Key type lgssc unregistered [ 3269.345217] LNet: 114672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3269.351408] LNetError: 114672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3269.360552] LNet: Removed LNI 192.168.206.117@tcp [ 3269.950206] Key type .llcrypt unregistered [ 3269.952993] Key type ._llcrypt unregistered [ 3289.906801] Key type ._llcrypt registered [ 3289.910544] Key type .llcrypt registered [ 3289.983582] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 3291.328012] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3291.349293] alg: No test for adler32 (adler32-zlib) [ 3292.424627] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 3292.710845] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 3294.383152] Key type lgssc registered [ 3295.378382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3302.966096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3303.000944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3304.434947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3308.747940] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3314.660533] LustreError: 115899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3314.676083] LustreError: 115899:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3316.453791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3316.759492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3322.001368] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3328.181632] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3334.354630] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3341.327512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3341.806971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3347.696151] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3348.983868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3354.648956] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3369.440553] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3369.454460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3369.471297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3371.492221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3371.495723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3371.509441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3371.526823] Lustre: Skipped 1 previous similar message [ 3376.608653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3376.628155] Lustre: Skipped 1 previous similar message [ 3381.731697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3381.735780] Lustre: Skipped 1 previous similar message [ 3382.240178] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3382.528176] Lustre: server umount lustre-OST0000 complete [ 3386.850646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3386.852710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3386.860330] Lustre: Skipped 1 previous similar message [ 3386.873066] Lustre: Skipped 1 previous similar message [ 3392.353153] Lustre: server umount lustre-MDT0000 complete [ 3395.880312] LustreError: 118499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832586 with bad export cookie 6883343943253093902 [ 3395.887227] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3395.895549] LustreError: 118499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3396.280646] Lustre: server umount lustre-MDT0001 complete [ 3406.523186] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 3409.014121] Key type lgssc unregistered [ 3409.336635] LNet: 119456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3409.343833] LNetError: 119456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3409.375320] LNet: Removed LNI 192.168.206.117@tcp [ 3410.425613] Key type .llcrypt unregistered [ 3410.429094] Key type ._llcrypt unregistered [ 3433.881661] Key type ._llcrypt registered [ 3433.885059] Key type .llcrypt registered [ 3434.003928] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 3434.695858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3434.714863] alg: No test for adler32 (adler32-zlib) [ 3436.040274] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 3436.346097] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 3438.039163] Key type lgssc registered [ 3439.677256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3447.936176] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3447.962049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3449.580685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3453.554310] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3459.042365] LustreError: 120872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3459.057064] LustreError: 120872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3461.192795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3461.620135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3466.004641] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3473.361492] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3478.892624] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3485.244963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3485.630219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3491.483776] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3495.870364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3498.665622] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3505.654177] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3508.212836] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3508.216419] Lustre: Skipped 2 previous similar messages [ 3511.413122] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3511.418568] Lustre: Skipped 2 previous similar messages [ 3512.381209] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3512.411903] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3517.785739] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.17@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3517.803097] Lustre: Skipped 2 previous similar messages [ 3522.724200] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.17@tcp 192.168.0.[2,10]@tcp [ 3527.689853] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.17/24@tcp in log lustre-MDT0000 [ 3527.696346] Lustre: Skipped 2 previous similar messages [ 3529.510400] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.17/24@tcp [ 3529.520970] Lustre: Skipped 1 previous similar message [ 3541.991191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3541.993670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3542.002400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3542.004782] LustreError: Skipped 1 previous similar message [ 3542.025666] Lustre: Skipped 1 previous similar message [ 3543.176433] Lustre: server umount lustre-OST0000 complete [ 3547.109069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3547.113342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3547.134635] Lustre: Skipped 1 previous similar message [ 3547.145313] Lustre: Skipped 2 previous similar messages [ 3549.400990] Lustre: server umount lustre-MDT0000 complete [ 3552.233069] LustreError: 120867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3552.266525] LustreError: 120867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3553.118387] LustreError: 120853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832744 with bad export cookie 18392843931448619989 [ 3553.123750] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3553.131250] LustreError: 120853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3553.460741] Lustre: server umount lustre-MDT0001 complete [ 3563.142716] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 3565.507585] Key type lgssc unregistered [ 3565.849448] LNet: 125626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3565.857410] LNetError: 125626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3566.888982] LNet: Removed LNI 192.168.206.117@tcp [ 3567.755245] Key type .llcrypt unregistered [ 3567.767159] Key type ._llcrypt unregistered [ 3592.059676] Key type ._llcrypt registered [ 3592.070696] Key type .llcrypt registered [ 3592.214412] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 3593.292849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3593.572773] alg: No test for adler32 (adler32-zlib) [ 3594.693606] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 3594.918567] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 3596.567130] Key type lgssc registered [ 3597.782371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3605.864663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3605.891818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3607.554851] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3607.597668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3607.616962] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.17/24@tcp [ 3612.382601] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3617.766278] LustreError: 127167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3617.791830] LustreError: 127167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3620.849536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3621.176059] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3621.220309] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.17/24@tcp [ 3621.259469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3625.335229] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3633.736176] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3640.274989] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3648.064135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3648.389505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3652.469186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3654.719188] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3661.540992] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3675.105421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3675.109066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3675.117081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3677.163056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3677.168120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3677.192260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3680.655039] Lustre: server umount lustre-OST0000 complete [ 3687.392682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3687.397273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3687.401221] Lustre: Skipped 1 previous similar message [ 3687.408899] Lustre: Skipped 2 previous similar messages [ 3690.353672] Lustre: server umount lustre-MDT0000 complete [ 3692.518210] LustreError: 127167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3692.536471] LustreError: 127167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3693.784595] LustreError: 127147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832884 with bad export cookie 8680064925908043033 [ 3693.794694] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3693.805697] LustreError: 127147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3694.106880] Lustre: server umount lustre-MDT0001 complete [ 3704.061148] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 3707.163917] Key type lgssc unregistered [ 3707.583898] LNet: 130759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3707.595628] LNetError: 130759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3707.615791] LNet: Removed LNI 192.168.206.117@tcp [ 3708.440557] Key type .llcrypt unregistered [ 3708.442459] Key type ._llcrypt unregistered [ 3732.616665] Key type ._llcrypt registered [ 3732.619662] Key type .llcrypt registered [ 3732.729686] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 3745.078874] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 3746.233645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3746.277312] alg: No test for adler32 (adler32-zlib) [ 3747.264253] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 3747.547069] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 3749.207238] Key type lgssc registered [ 3750.328616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3782.861626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3782.880364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3784.045118] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3784.066393] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3784.119295] Lustre: lustre-MDT0000: new disk, initializing [ 3784.177130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3784.188523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3787.887180] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3798.046820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3798.157467] Lustre: 134140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3798.230156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3798.234812] Lustre: Skipped 1 previous similar message [ 3798.329623] Lustre: lustre-MDT0001: new disk, initializing [ 3798.417721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3798.485938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3798.503639] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3802.182935] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3811.597902] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3817.224276] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3823.909371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3824.109323] Lustre: lustre-OST0000: new disk, initializing [ 3824.116156] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3824.129864] Lustre: 136110:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3824.205983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3830.685558] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3834.422802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3834.429171] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3834.545851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3839.454628] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3844.662169] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3844.822095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3849.825638] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3850.008160] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3855.331693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3855.333581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3855.347844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3855.397134] Lustre: Skipped 1 previous similar message [ 3857.911215] Lustre: server umount lustre-OST0000 complete [ 3865.059215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3865.070135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3865.090898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3865.095441] Lustre: Skipped 1 previous similar message [ 3867.701698] Lustre: server umount lustre-MDT0000 complete [ 3871.364416] LustreError: 134135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833062 with bad export cookie 3897454217045121088 [ 3871.388114] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3871.833708] Lustre: server umount lustre-MDT0001 complete [ 3879.162102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3879.781060] LustreError: 138325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3879.813534] LustreError: 138325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3879.900796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3884.087286] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3885.026529] LustreError: 138326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3889.127788] LustreError: 138325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3892.848482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3893.131807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3897.013241] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3904.441771] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3910.129446] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3918.600397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3918.768262] LustreError: 140146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3918.969712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3925.552427] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3932.241794] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3936.056717] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3936.299408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3940.511644] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3940.739051] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3949.492089] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3949.630171] Lustre: lustre-OST0001: new disk, initializing [ 3949.633116] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3949.651416] Lustre: 141698:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3949.782834] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3956.109138] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3957.390982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3957.416656] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3957.573282] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3965.197533] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3965.389897] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3969.575919] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3969.770704] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3975.408432] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9414c6c11800.ost_server_uuid 50 [ 3976.994464] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9414c6c11800.ost_server_uuid in FULL state after 0 sec [ 3988.459581] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3988.476910] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3988.484080] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3988.489930] Lustre: Skipped 2 previous similar messages [ 3988.498383] Lustre: Skipped 1 previous similar message [ 3990.518667] Lustre: server umount lustre-OST0001 complete [ 3993.571729] LustreError: 140445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3993.599034] LustreError: 140445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3996.129541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3996.136119] LustreError: Skipped 1 previous similar message [ 3996.139391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3996.153846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3996.159831] Lustre: Skipped 2 previous similar messages [ 4000.020794] Lustre: server umount lustre-OST0000 complete [ 4003.809279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4003.822265] Lustre: Skipped 1 previous similar message [ 4003.831657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4003.842493] Lustre: Skipped 3 previous similar messages [ 4009.609311] Lustre: server umount lustre-MDT0000 complete [ 4012.893674] LustreError: 138305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833203 with bad export cookie 3897454217045121970 [ 4012.901373] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4012.906382] LustreError: 138305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4013.218433] Lustre: server umount lustre-MDT0001 complete [ 4030.969519] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 4033.542324] Key type lgssc unregistered [ 4033.836140] LNet: 144702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4033.851305] LNetError: 144702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4033.883843] LNet: Removed LNI 192.168.206.117@tcp [ 4034.645940] Key type .llcrypt unregistered [ 4034.650729] Key type ._llcrypt unregistered [ 4071.704415] Key type ._llcrypt registered [ 4071.708986] Key type .llcrypt registered [ 4071.803302] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 4085.045148] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 4086.235329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4086.258460] alg: No test for adler32 (adler32-zlib) [ 4087.277641] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 4087.538727] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 4089.207177] Key type lgssc registered [ 4090.347185] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4126.472814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4126.503324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4127.938639] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4128.011900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4128.082470] Lustre: lustre-MDT0000: new disk, initializing [ 4128.196301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4128.243403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4133.521337] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4143.559894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4143.691741] Lustre: 148838:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4143.753583] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4143.759910] Lustre: Skipped 1 previous similar message [ 4143.830370] Lustre: lustre-MDT0001: new disk, initializing [ 4143.928622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4143.967772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4143.976700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4147.920298] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4156.804970] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4162.062711] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4168.020895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4168.303987] Lustre: lustre-OST0000: new disk, initializing [ 4168.308328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4168.315151] Lustre: 150804:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4168.404508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4174.427356] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4176.443284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4176.459847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4176.536730] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4183.561270] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4188.479796] Lustre: Failing over lustre-OST0000 [ 4188.681820] Lustre: server umount lustre-OST0000 complete [ 4206.818905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4207.008352] Lustre: 147062:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787833382/real 1787833382] req@00000000ba996255 x1874679067345536/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787833398 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4207.043629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4207.063177] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4207.080416] Lustre: Skipped 1 previous similar message [ 4207.210945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4207.231048] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4207.877615] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4208.095466] Lustre: 147060:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787833383/real 1787833383] req@000000005a97f528 x1874679067346304/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787833399 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4209.120606] Lustre: 147059:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787833383/real 1787833383] req@00000000ed280f5e x1874679067346688/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787833399 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4209.148218] Lustre: 147059:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4212.224427] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4212.227654] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4212.604881] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4213.281911] Lustre: 147060:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787833388/real 1787833388] req@0000000022c61536 x1874679067347712/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787833404 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4215.930401] Lustre: Failing over lustre-MDT0000 [ 4216.168875] Lustre: server umount lustre-MDT0000 complete [ 4217.321956] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4217.337822] Lustre: Skipped 3 previous similar messages [ 4218.400305] Lustre: 147060:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787833393/real 1787833393] req@00000000857c1674 x1874679067348864/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787833409 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4218.417957] Lustre: 147060:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4222.432815] LustreError: 150812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4222.453038] LustreError: 150812:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4223.175219] LustreError: 148843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4227.553428] LustreError: 148848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4227.579838] LustreError: 148848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4232.166116] Lustre: 148840:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787833407/real 1787833407] req@000000003c9904a4 x1874679067356672/t0(0) o101->MGC192.168.206.117@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1787833423 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4232.187235] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4232.196555] LustreError: 148849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4232.203496] Lustre: 148840:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.117@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4232.206060] LustreError: 148849:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4233.657335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4237.286499] LustreError: 150812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4237.313512] LustreError: 150812:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4242.660619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4242.712706] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4243.662950] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4247.315827] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4248.054605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4248.060532] Lustre: Skipped 1 previous similar message [ 4248.082094] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4248.147235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4258.282361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4258.284038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4258.295987] Lustre: Skipped 1 previous similar message [ 4258.311780] Lustre: Skipped 1 previous similar message [ 4261.046428] Lustre: server umount lustre-OST0000 complete [ 4268.514584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4268.515911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4268.544230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4268.567922] Lustre: Skipped 1 previous similar message [ 4270.764534] Lustre: server umount lustre-MDT0000 complete [ 4273.637711] LustreError: 150812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4273.663736] LustreError: 150812:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4275.109393] LustreError: 148829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833466 with bad export cookie 8225674049575656744 [ 4275.117858] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4275.126906] LustreError: 148829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4275.543372] Lustre: server umount lustre-MDT0001 complete [ 4286.705399] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 4289.239791] Key type lgssc unregistered [ 4289.496580] LNet: 154469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4289.506098] LNetError: 154469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4289.524952] LNet: Removed LNI 192.168.206.117@tcp [ 4290.170066] Key type .llcrypt unregistered [ 4290.172040] Key type ._llcrypt unregistered [ 4323.861596] Key type ._llcrypt registered [ 4323.865421] Key type .llcrypt registered [ 4323.957598] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 4324.801631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4324.819832] alg: No test for adler32 (adler32-zlib) [ 4325.943659] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 4326.202092] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 4327.911299] Key type lgssc registered [ 4328.886634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4335.257082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4335.281364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4336.670400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4340.759702] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4346.848174] LustreError: 155988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4346.871067] LustreError: 155988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4349.226222] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4351.978210] LustreError: 155987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4352.556205] LustreError: 156016:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4352.561840] LustreError: 156016:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4352.568749] LustreError: 156016:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 4352.779793] Lustre: server umount lustre-MDT0000 complete [ 4366.805547] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 4375.732059] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 4407.260683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4407.449463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4407.469808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4407.539631] Lustre: lustre-MDT0000: new disk, initializing [ 4407.608975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4407.624191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4411.985930] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4422.865858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4422.950572] Lustre: 159320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4423.045339] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4423.053096] Lustre: Skipped 1 previous similar message [ 4423.113478] Lustre: lustre-MDT0001: new disk, initializing [ 4423.193914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4423.242770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4423.254995] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4428.257139] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4436.517245] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4441.114191] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4447.457562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4447.695555] Lustre: lustre-OST0000: new disk, initializing [ 4447.700962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4447.708464] Lustre: 161286:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4447.792862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4453.924903] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4457.585937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4457.609435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4457.724183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4463.563140] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4468.389375] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4470.430778] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 08:27:40 (1787833660) [ 4720.095839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4720.100561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4720.110840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4722.144990] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4722.145664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4722.164511] Lustre: Skipped 1 previous similar message [ 4722.183045] Lustre: Skipped 2 previous similar messages [ 4726.400658] Lustre: server umount lustre-MDT0000 complete [ 4727.263856] LustreError: 159330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4727.282221] LustreError: 159330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4729.759403] LustreError: 159311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833920 with bad export cookie 8938469656535640158 [ 4729.780346] LustreError: 159311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4729.783204] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4732.388204] LustreError: 163072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4732.389108] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4732.427816] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4734.434085] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4739.551758] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4739.551758] LustreError: 159327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4739.551767] LustreError: 159327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4744.159608] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4744.448844] Lustre: server umount lustre-MDT0001 complete [ 4763.103156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4763.307372] Lustre: server umount lustre-OST0000 complete [ 4773.775787] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 4785.909896] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 4820.687554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4820.955992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4820.976745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4821.067599] Lustre: lustre-MDT0000: new disk, initializing [ 4821.158758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4821.190211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4825.567589] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4836.497766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4836.730055] Lustre: 165962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4836.795165] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4836.799780] Lustre: Skipped 1 previous similar message [ 4836.893480] Lustre: lustre-MDT0001: new disk, initializing [ 4837.003232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4837.044252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4837.054128] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4841.736567] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4851.371500] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4857.519553] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4865.100816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4865.442779] Lustre: lustre-OST0000: new disk, initializing [ 4865.445886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4865.456527] Lustre: 167929:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4865.555968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4872.220582] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4873.767437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4873.780595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4873.913979] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4882.679981] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4887.589637] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4887.850229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4892.630887] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4892.845312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4899.296437] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4899.302494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4899.323858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4909.023126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4909.240409] Lustre: server umount lustre-OST0000 complete [ 4913.633681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4913.639545] LustreError: Skipped 1 previous similar message [ 4913.642375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4913.652604] Lustre: Skipped 1 previous similar message [ 4919.293472] Lustre: server umount lustre-MDT0000 complete [ 4920.288531] LustreError: 165972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4920.319802] LustreError: 165972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4923.055855] LustreError: 165953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834114 with bad export cookie 8938469656535798988 [ 4923.057834] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4923.067307] LustreError: 165953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4923.604192] Lustre: server umount lustre-MDT0001 complete [ 4932.500913] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:35:22 (1787834122) [ 4947.761543] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 4957.127425] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 4990.898429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4991.168843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4991.208503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4991.298458] Lustre: lustre-MDT0000: new disk, initializing [ 4991.372417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4991.391239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4995.890734] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5006.369959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5006.474300] Lustre: 172684:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5006.756994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5011.338990] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5020.074500] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5025.691046] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5032.790174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5033.200326] Lustre: lustre-OST0000: new disk, initializing [ 5033.202699] Lustre: Skipped 1 previous similar message [ 5033.205692] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5033.209222] Lustre: Skipped 2 previous similar messages [ 5033.213913] Lustre: 174662:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5039.735985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5039.741136] Lustre: Skipped 1 previous similar message [ 5039.745740] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5039.863758] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5039.930337] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5048.510397] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5055.456528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5055.465486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5055.479836] Lustre: Skipped 1 previous similar message [ 5055.492651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5055.496611] Lustre: Skipped 6 previous similar messages [ 5061.767714] Lustre: server umount lustre-OST0000 complete [ 5071.701460] Lustre: server umount lustre-MDT0000 complete [ 5075.159735] LustreError: 172676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834266 with bad export cookie 8938469656535799870 [ 5075.166770] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5075.169837] LustreError: 172676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5075.514747] Lustre: server umount lustre-MDT0001 complete [ 5083.329452] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:37:53 (1787834273) [ 5099.308659] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 5109.040154] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 5149.045560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5149.368409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5149.451566] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5149.535469] Lustre: lustre-MDT0000: new disk, initializing [ 5149.588780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5149.591629] Lustre: Skipped 2 previous similar messages [ 5149.600959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5153.304704] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5163.931901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5164.059864] Lustre: 178985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5164.289220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5168.908690] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5177.475238] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5182.757914] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5188.226984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5188.459346] Lustre: 180961:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5193.634343] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5197.417643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5197.596085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5202.325934] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5213.157626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5213.159236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5213.178501] LustreError: Skipped 2 previous similar messages [ 5213.186370] Lustre: Skipped 3 previous similar messages [ 5213.191067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5213.201531] Lustre: Skipped 8 previous similar messages [ 5215.959659] Lustre: server umount lustre-OST0000 complete [ 5225.955619] Lustre: server umount lustre-MDT0000 complete [ 5228.515814] LustreError: 179867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5228.535123] LustreError: 179867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5229.342247] LustreError: 180960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834420 with bad export cookie 8938469656535800955 [ 5229.348742] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5229.361092] LustreError: 180960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5229.667357] Lustre: server umount lustre-MDT0001 complete [ 5241.062261] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 5244.467495] Key type lgssc unregistered [ 5244.819610] LNet: 182819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5244.826805] LNetError: 182819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5244.837448] LNet: Removed LNI 192.168.206.117@tcp [ 5245.567202] Key type .llcrypt unregistered [ 5245.570227] Key type ._llcrypt unregistered [ 5271.853471] Key type ._llcrypt registered [ 5271.855728] Key type .llcrypt registered [ 5272.007819] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 5272.715938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5272.879380] alg: No test for adler32 (adler32-zlib) [ 5274.041709] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 5274.287372] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 5276.104186] Key type lgssc registered [ 5277.834844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5286.664720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5286.688171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5288.122176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5292.471937] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5298.660937] LustreError: 184238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5298.676326] LustreError: 184238:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5300.453271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5300.764420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5304.936117] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5311.588315] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5316.879776] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5323.775991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5324.218200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5330.711030] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5333.504692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5338.155567] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5359.073606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5359.077646] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5359.110159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5359.126869] Lustre: Skipped 1 previous similar message [ 5362.203658] Lustre: server umount lustre-OST0000 complete [ 5367.266113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5367.271738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5367.277323] Lustre: Skipped 1 previous similar message [ 5367.282443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5369.317110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5369.321984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5371.835658] Lustre: server umount lustre-MDT0000 complete [ 5374.439839] LustreError: 184238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5375.084884] LustreError: 184220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834566 with bad export cookie 12363465916714600882 [ 5375.091044] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5375.099220] LustreError: 184220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5375.398748] Lustre: server umount lustre-MDT0001 complete [ 5384.933515] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 5388.354867] Key type lgssc unregistered [ 5388.693944] LNet: 187793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5388.711157] LNetError: 187793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5388.733537] LNet: Removed LNI 192.168.206.117@tcp [ 5389.647113] Key type .llcrypt unregistered [ 5389.649212] Key type ._llcrypt unregistered [ 5412.394855] Key type ._llcrypt registered [ 5412.396604] Key type .llcrypt registered [ 5412.479332] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 5413.522696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5413.533480] alg: No test for adler32 (adler32-zlib) [ 5414.553392] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 5414.797875] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 5416.496349] Key type lgssc registered [ 5417.863708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5425.323931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5425.342568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5426.752962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5431.410993] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5436.902526] LustreError: 189210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5436.913397] LustreError: 189210:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5439.514128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5439.960408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5444.139334] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5451.377925] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5457.835250] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5465.524087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5465.950585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5471.099389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5472.833160] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5481.095229] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5491.690349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5491.702180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5491.729517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5493.992902] Lustre: lustre-OST0000: Not available for connect from 192.168.206.17@tcp (stopping) [ 5494.767955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5494.769854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5497.282779] Lustre: server umount lustre-OST0000 complete [ 5504.609791] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9414d2149800.ost_server_uuid 50 [ 5506.066865] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5519.331786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5519.345618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5519.353853] Lustre: Skipped 2 previous similar messages [ 5524.448766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5524.455294] Lustre: Skipped 2 previous similar messages [ 5525.669978] Lustre: server umount lustre-MDT0000 complete [ 5529.257943] LustreError: 192256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834720 with bad export cookie 17065303462334254352 [ 5529.259749] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5529.269272] LustreError: 192256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5529.619426] Lustre: server umount lustre-MDT0001 complete [ 5538.039994] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 08:45:28 (1787834728) [ 5543.963386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5544.339565] LustreError: 193128:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5544.395625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5549.207823] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5549.547290] LustreError: 193129:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5554.660513] LustreError: 193128:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5559.085693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5559.499196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5564.671224] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5573.357893] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5579.646116] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5586.819615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5587.189954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5589.243676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5593.520419] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5601.544495] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5608.408759] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5608.652187] Lustre: lustre-OST0001: new disk, initializing [ 5608.658254] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5608.664154] Lustre: 195969:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5608.755952] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5615.929897] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5617.859260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5617.890650] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5618.056852] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5627.684229] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5633.508868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5633.521042] Lustre: Skipped 1 previous similar message [ 5633.530627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5633.537270] Lustre: Skipped 1 previous similar message [ 5636.894708] Lustre: server umount lustre-OST0000 complete [ 5638.631169] LustreError: 194951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5638.649059] LustreError: 194951:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5643.060511] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5643.279190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5647.884813] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5648.126883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5648.870853] LustreError: 194949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5648.894526] LustreError: 194949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5664.229207] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5664.238990] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5664.240272] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5664.240278] Lustre: Skipped 1 previous similar message [ 5664.240480] Lustre: Skipped 2 previous similar messages [ 5664.263147] LustreError: Skipped 1 previous similar message [ 5667.974508] Lustre: server umount lustre-OST0001 complete [ 5671.907695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5674.465612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5674.475494] Lustre: Skipped 1 previous similar message [ 5677.682036] Lustre: server umount lustre-MDT0000 complete [ 5679.585877] LustreError: 193125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5679.605937] LustreError: 193125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 5681.045940] LustreError: 195968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834872 with bad export cookie 17065303462334255423 [ 5681.048631] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5681.054056] LustreError: 195968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5681.397335] Lustre: server umount lustre-MDT0001 complete [ 5702.131264] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 08:48:12 (1787834892) [ 5707.683780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5708.004675] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5708.046518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5708.240987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5711.865395] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5718.805297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5718.877458] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5718.914895] Lustre: 199407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5722.833944] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5729.503552] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5734.502215] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5739.895813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5740.018438] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5744.760663] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5746.818733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5750.910142] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5756.743086] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5756.790520] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5762.358427] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5768.829226] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5771.753334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5771.756355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5771.768847] Lustre: Skipped 1 previous similar message [ 5771.775919] Lustre: Skipped 4 previous similar messages [ 5777.379867] Lustre: server umount lustre-OST0000 complete [ 5779.656906] LustreError: 201258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5797.354872] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5797.367192] LustreError: Skipped 1 previous similar message [ 5799.272841] Lustre: server umount lustre-OST0001 complete [ 5807.585358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5807.595459] Lustre: Skipped 9 previous similar messages [ 5808.686988] Lustre: server umount lustre-MDT0000 complete [ 5812.629496] LustreError: 201277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835003 with bad export cookie 17065303462334256830 [ 5812.636090] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5812.647878] LustreError: 201277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5813.082963] Lustre: server umount lustre-MDT0001 complete [ 5837.112261] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 08:50:27 (1787835027) [ 5852.053107] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 5860.476509] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 5896.162271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5896.338618] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5896.360454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5896.430579] Lustre: lustre-MDT0000: new disk, initializing [ 5896.515098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5896.524534] Lustre: Skipped 3 previous similar messages [ 5896.537275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5900.245578] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5909.533397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5909.651209] Lustre: 207509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5909.694958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5909.699108] Lustre: Skipped 1 previous similar message [ 5909.750457] Lustre: lustre-MDT0001: new disk, initializing [ 5909.844484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5909.858921] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5914.427651] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5925.441944] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5931.874511] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5939.053396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5939.277728] Lustre: lustre-OST0000: new disk, initializing [ 5939.281566] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5939.288391] Lustre: 209476:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5945.285365] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5945.452610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5945.466337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5945.524464] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5954.552428] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5960.917366] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5961.073848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5966.017610] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5966.235882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5971.435734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5971.441386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5971.441877] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5971.456945] Lustre: Skipped 5 previous similar messages [ 5971.501552] Lustre: Skipped 1 previous similar message [ 5983.199163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5983.307696] Lustre: server umount lustre-OST0000 complete [ 5992.819185] Lustre: server umount lustre-MDT0000 complete [ 5997.020496] LustreError: 207499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835188 with bad export cookie 17065303462334258391 [ 5997.020906] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5997.036840] LustreError: 207499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5997.318889] Lustre: server umount lustre-MDT0001 complete [ 6004.956588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6005.348534] LustreError: 211694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6005.364407] LustreError: 211694:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6009.661315] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6017.625343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6022.147485] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6030.461862] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6035.904372] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6041.638973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6041.859470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6041.863974] Lustre: Skipped 4 previous similar messages [ 6047.537454] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6054.105601] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6058.378689] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6058.578967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6062.421178] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6062.592111] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6065.633977] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6065.641862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6065.650553] LustreError: Skipped 2 previous similar messages [ 6065.668089] Lustre: Skipped 2 previous similar messages [ 6070.419352] Lustre: server umount lustre-OST0000 complete [ 6075.876013] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6079.925226] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6109.070411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6115.814805] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6123.374165] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6127.886156] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6130.921095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.932717] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.939942] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6130.963800] Lustre: Skipped 1 previous similar message [ 6136.288217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6136.294327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6136.299336] LustreError: Skipped 1 previous similar message [ 6136.310664] Lustre: Skipped 11 previous similar messages [ 6137.551029] Lustre: server umount lustre-OST0000 complete [ 6155.231441] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6155.427451] Lustre: server umount lustre-MDT0000 complete [ 6156.769354] LustreError: 211695:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6156.775745] LustreError: 211695:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6158.931735] LustreError: 211676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835350 with bad export cookie 17065303462334259273 [ 6158.939654] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6158.940458] LustreError: 211676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6159.188741] Lustre: server umount lustre-MDT0001 complete [ 6171.388865] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 08:56:00 (1787835360) [ 6180.959592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6187.528310] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6194.587635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6198.669564] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6204.940894] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6211.225371] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6219.068581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6228.537483] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6236.799451] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6240.902943] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6241.110421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6245.865482] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6246.205151] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6252.695559] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6252.873721] Lustre: lustre-OST0001: new disk, initializing [ 6252.876655] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6252.882390] Lustre: 221033:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6258.838680] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6262.865769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6262.873367] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6262.962133] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6267.949146] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6272.255287] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6272.430315] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6276.503224] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6276.763845] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6278.628958] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6278.631435] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6278.638461] LustreError: Skipped 1 previous similar message [ 6278.661399] Lustre: Skipped 5 previous similar messages [ 6284.769977] Lustre: server umount lustre-OST0001 complete [ 6290.627767] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6290.874535] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6294.900369] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6295.115994] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6322.544917] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6322.716898] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6322.723105] Lustre: Skipped 5 previous similar messages [ 6327.783287] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6327.800091] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6327.803784] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6327.821024] Lustre: Skipped 1 previous similar message [ 6328.011752] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6334.294565] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6335.872796] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6352.351248] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6352.420207] Lustre: server umount lustre-OST0001 complete [ 6383.071147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6383.265424] Lustre: server umount lustre-OST0000 complete [ 6393.316511] Lustre: server umount lustre-MDT0000 complete [ 6397.053405] LustreError: 217641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835588 with bad export cookie 17065303462334260239 [ 6397.059078] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6397.066486] LustreError: 217641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6403.567642] Lustre: server umount lustre-MDT0001 complete [ 6426.188129] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 09:00:16 (1787835616) [ 6432.217154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6432.418069] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6432.491353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6436.579487] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6445.711544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6445.857556] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6445.891296] Lustre: 226278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6451.022816] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6458.917766] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6464.918772] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6471.203340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6471.374693] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6477.231399] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6483.517616] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6492.746035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6492.835630] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6499.187629] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6507.954346] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6509.587833] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6514.367611] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6514.605146] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6519.738566] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6520.001544] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6526.862638] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9414c38d4800.ost_server_uuid 50 [ 6529.344460] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9414c38d4800.ost_server_uuid in IDLE state after 0 sec [ 6530.439912] Lustre: Permanently deactivating lustre-OST0001 [ 6530.460510] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6534.307410] Lustre: Permanently reactivating lustre-OST0001 [ 6540.783428] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6540.814779] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6540.815367] Lustre: Skipped 7 previous similar messages [ 6540.820056] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6540.820063] LustreError: Skipped 5 previous similar messages [ 6540.857308] Lustre: Skipped 29 previous similar messages [ 6541.133296] Lustre: server umount lustre-OST0001 complete [ 6545.895170] LustreError: 228621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6545.919238] LustreError: 228621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [ 6552.731078] Lustre: server umount lustre-OST0000 complete [ 6562.356200] Lustre: server umount lustre-MDT0000 complete [ 6565.713602] LustreError: 226268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835756 with bad export cookie 17065303462334261527 [ 6565.719300] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6565.720927] LustreError: 226268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6566.134239] Lustre: server umount lustre-MDT0001 complete [ 6588.573758] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 09:02:58 (1787835778) [ 6596.354034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6596.916958] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6596.982568] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6596.999302] Lustre: Skipped 5 previous similar messages [ 6602.572677] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6612.584170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6612.711309] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6612.784427] Lustre: 232981:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6617.299022] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6624.504949] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6629.683423] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6636.315321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6636.525472] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6636.541815] Lustre: Permanently deactivating lustre-OST0000 [ 6636.549456] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6642.011750] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6645.862579] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6650.402510] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6658.759455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6658.904502] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6665.628175] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6668.031560] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6673.755586] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6675.569361] Lustre: Permanently reactivating lustre-OST0000 [ 6675.581990] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6675.587405] Lustre: Skipped 2 previous similar messages [ 6676.385700] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6676.394629] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6676.403675] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6676.486177] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6676.516295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6698.969984] Lustre: server umount lustre-OST0001 complete [ 6708.574775] Lustre: server umount lustre-OST0000 complete [ 6718.503439] Lustre: server umount lustre-MDT0000 complete [ 6722.280405] LustreError: 233724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835913 with bad export cookie 17065303462334263200 [ 6722.288759] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6722.312738] LustreError: 233724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6722.774196] Lustre: server umount lustre-MDT0001 complete [ 6733.584374] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 6736.293389] Key type lgssc unregistered [ 6736.598690] LNet: 237867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6736.611429] LNetError: 237867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6736.631565] LNet: Removed LNI 192.168.206.117@tcp [ 6737.443146] Key type .llcrypt unregistered [ 6737.448620] Key type ._llcrypt unregistered [ 6757.642299] Key type ._llcrypt registered [ 6757.645868] Key type .llcrypt registered [ 6757.759962] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 6758.746059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6758.817153] alg: No test for adler32 (adler32-zlib) [ 6759.927225] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 6760.158368] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 6761.856782] Key type lgssc registered [ 6762.817791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6773.070328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6773.080716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6774.438159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6777.298651] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6784.057515] LustreError: 239314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6784.075748] LustreError: 239314:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6788.534555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6788.634814] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6788.656428] Lustre: 239303:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6788.663799] Lustre: Permanently deactivating lustre-MDT0001 [ 6788.678073] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6789.072742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6791.029737] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6791.051306] LustreError: 239345:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6791.055819] LustreError: 239345:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6791.060952] LustreError: 239345:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 6794.059647] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6803.154158] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6809.520246] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6817.366439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6817.634817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6819.695899] LustreError: 241154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6819.706758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6824.513110] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6824.937993] LustreError: 241278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6830.048563] LustreError: 241154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6832.786118] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6833.130319] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6838.249262] LustreError: 241157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6838.256814] LustreError: 241157:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6838.897987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6839.190191] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6843.764655] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6844.413857] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6844.418268] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6850.245642] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6852.972516] Lustre: Permanently reactivating lustre-MDT0001 [ 6852.983508] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6852.995917] Lustre: Skipped 2 previous similar messages [ 6857.010336] Lustre: lustre-MDT0001: Client 9293dfd4-3749-4ea3-97ee-3b0d3086e7e5 (at 192.168.206.17@tcp) reconnecting [ 6862.399173] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6864.238936] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6869.975955] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6870.214283] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6876.776390] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6877.051893] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6880.191916] Lustre: Permanently deactivating lustre-MDT0001 [ 6880.212935] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6880.220425] Lustre: Skipped 2 previous similar messages [ 6886.568481] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6895.587247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6895.593089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6895.600249] Lustre: Skipped 2 previous similar messages [ 6895.614289] Lustre: Skipped 3 previous similar messages [ 6896.281897] Lustre: server umount lustre-MDT0000 complete [ 6899.622843] LustreError: 239294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836090 with bad export cookie 17866571438475958266 [ 6899.625906] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6899.633329] LustreError: 239294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6899.885302] Lustre: server umount lustre-MDT0001 complete [ 6902.975226] Lustre: server umount lustre-OST0000 complete [ 6912.647059] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 09:08:23 (1787836103) [ 6922.699623] Lustre: server umount lustre-OST0001 complete [ 6929.388260] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 6939.306730] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 6981.757418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6982.043972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6982.052471] Lustre: Skipped 2 previous similar messages [ 6982.087671] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6982.181759] Lustre: lustre-MDT0000: new disk, initializing [ 6982.253407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6982.269814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6987.563309] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6999.086499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6999.202599] Lustre: 247572:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6999.318841] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6999.328107] Lustre: Skipped 1 previous similar message [ 6999.386179] Lustre: lustre-MDT0001: new disk, initializing [ 6999.591310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6999.684391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6999.709268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7003.302560] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7013.403861] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7020.054733] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7028.012298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7028.586796] Lustre: lustre-OST0000: new disk, initializing [ 7028.598449] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7028.615700] Lustre: 249540:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7028.747479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7034.441077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7034.453550] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7034.585956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7035.037625] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7044.105454] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7046.286429] LustreError: 247577:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7054.476578] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7054.692494] Lustre: lustre-OST0001: new disk, initializing [ 7054.696401] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7054.703805] Lustre: 250658:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7056.335110] LustreError: 247577:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7056.362158] LustreError: 247578:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7061.099270] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7064.119808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7064.136888] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7064.206178] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7066.465667] LustreError: 247578:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7069.657423] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7071.214395] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7074.291695] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7074.296878] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7074.302361] Lustre: Skipped 2 previous similar messages [ 7074.312742] Lustre: Skipped 1 previous similar message [ 7076.062313] Lustre: lustre-OST0001: Not available for connect from 192.168.206.17@tcp (stopping) [ 7079.104744] Lustre: server umount lustre-OST0001 complete [ 7079.404060] LustreError: 249856:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7096.288819] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7096.291309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7096.300911] LustreError: Skipped 1 previous similar message [ 7096.302702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7096.321590] Lustre: Skipped 1 previous similar message [ 7096.335930] Lustre: Skipped 1 previous similar message [ 7099.878356] LustreError: 249526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7099.903261] LustreError: 249526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7101.839084] Lustre: server umount lustre-OST0000 complete [ 7107.044179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7107.049647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7107.063695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7107.067604] Lustre: Skipped 2 previous similar messages [ 7111.841297] Lustre: server umount lustre-MDT0000 complete [ 7115.267270] LustreError: 247563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836306 with bad export cookie 17866571438475961087 [ 7115.267562] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7115.274987] LustreError: 247563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7121.768887] Lustre: server umount lustre-MDT0001 complete [ 7131.740691] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 7134.254279] Key type lgssc unregistered [ 7134.570781] LNet: 252691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7134.580355] LNetError: 252691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7134.597613] LNet: Removed LNI 192.168.206.117@tcp [ 7135.324525] Key type .llcrypt unregistered [ 7135.327792] Key type ._llcrypt unregistered [ 7175.164656] Key type ._llcrypt registered [ 7175.167289] Key type .llcrypt registered [ 7175.299614] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 7176.350233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7176.376524] alg: No test for adler32 (adler32-zlib) [ 7177.633981] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 7177.945626] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 7179.631142] Key type lgssc registered [ 7180.601860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7188.447301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7188.476820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7189.834374] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7189.871797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7190.081155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7195.542526] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7206.221482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7206.352646] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7206.442427] Lustre: 254902:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7206.735256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7211.868412] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7220.233329] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7227.187243] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7236.336870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7236.554076] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7236.743985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7243.984597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7244.323277] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7252.896213] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7305.699714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7305.706496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7305.707227] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7305.716820] Lustre: Skipped 1 previous similar message [ 7305.744484] Lustre: Skipped 1 previous similar message [ 7310.818751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7310.833621] Lustre: Skipped 1 previous similar message [ 7311.463238] Lustre: server umount lustre-OST0000 complete [ 7316.035154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7329.145514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7329.523989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7335.952307] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7336.683074] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7336.694655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7336.709279] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7336.740547] Lustre: Skipped 1 previous similar message [ 7343.904846] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7383.025859] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7383.040776] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7383.050764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7383.055596] Lustre: Skipped 1 previous similar message [ 7388.145653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7388.154363] Lustre: Skipped 1 previous similar message [ 7388.871580] Lustre: server umount lustre-OST0000 complete [ 7393.253026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7393.263571] Lustre: Skipped 1 previous similar message [ 7393.267618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7393.272397] Lustre: Skipped 1 previous similar message [ 7398.830069] Lustre: server umount lustre-MDT0000 complete [ 7402.679940] LustreError: 254893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836593 with bad export cookie 6359111993995636812 [ 7402.683024] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7402.693282] LustreError: 254893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7403.081464] Lustre: server umount lustre-MDT0001 complete [ 7413.848434] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 7416.736674] Key type lgssc unregistered [ 7417.055462] LNet: 260331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7417.062269] LNetError: 260331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7417.090869] LNet: Removed LNI 192.168.206.117@tcp [ 7417.817749] Key type .llcrypt unregistered [ 7417.820089] Key type ._llcrypt unregistered [ 7441.646963] Key type ._llcrypt registered [ 7441.649405] Key type .llcrypt registered [ 7441.747267] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 7442.824820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7442.870585] alg: No test for adler32 (adler32-zlib) [ 7443.945718] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 7444.225877] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 7445.928049] Key type lgssc registered [ 7447.028940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7454.136518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7454.157308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7455.532988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7460.158467] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7464.932147] LustreError: 261748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7464.953357] LustreError: 261748:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7468.822700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7469.255694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7473.496229] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7481.384183] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7488.042801] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7494.358333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7494.620143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7500.786710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7500.866648] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7507.949516] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7534.567330] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7534.579604] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7534.601631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7534.611607] Lustre: Skipped 1 previous similar message [ 7539.430790] Lustre: server umount lustre-OST0000 complete [ 7544.802679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7544.808511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7544.813778] Lustre: Skipped 2 previous similar messages [ 7549.182797] Lustre: server umount lustre-MDT0000 complete [ 7549.923904] LustreError: 261748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7549.955356] LustreError: 261748:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7552.581915] LustreError: 261727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836743 with bad export cookie 4892587715625400162 [ 7552.593153] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7552.596619] LustreError: 261727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7553.001048] Lustre: server umount lustre-MDT0001 complete [ 7563.491161] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 7566.539692] Key type lgssc unregistered [ 7566.856783] LNet: 266056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7566.867183] LNetError: 266056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7566.883782] LNet: Removed LNI 192.168.206.117@tcp [ 7567.765348] Key type .llcrypt unregistered [ 7567.771084] Key type ._llcrypt unregistered [ 7585.964622] Key type ._llcrypt registered [ 7585.967540] Key type .llcrypt registered [ 7586.082669] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 7587.190950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7587.225768] alg: No test for adler32 (adler32-zlib) [ 7588.342501] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 7588.720573] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 7590.431204] Key type lgssc registered [ 7591.819096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7599.875864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7599.889356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7601.402317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7606.131770] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7611.879440] LustreError: 267277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7611.901124] LustreError: 267277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7615.134448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7615.550326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7620.297521] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7628.158969] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7636.193615] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7643.322956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7643.816648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7646.903991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7650.490849] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7658.978596] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7670.765187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7670.769230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7670.790621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7670.801960] Lustre: Skipped 1 previous similar message [ 7675.872811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7675.890224] Lustre: Skipped 1 previous similar message [ 7676.930136] Lustre: server umount lustre-OST0000 complete [ 7681.011522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7681.017225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7681.037792] Lustre: Skipped 2 previous similar messages [ 7681.047591] Lustre: Skipped 1 previous similar message [ 7686.114748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7686.129559] Lustre: Skipped 1 previous similar message [ 7691.233872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7691.243023] Lustre: Skipped 1 previous similar message [ 7695.329602] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7695.555141] Lustre: server umount lustre-MDT0000 complete [ 7696.353655] LustreError: 268517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7696.367994] LustreError: 268517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7700.249641] LustreError: 267257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836891 with bad export cookie 14784731960064833581 [ 7700.253579] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7700.268106] LustreError: 267257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7700.645552] Lustre: server umount lustre-MDT0001 complete [ 7712.001530] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 7714.168367] Key type lgssc unregistered [ 7714.379942] LNet: 270960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7714.397575] LNetError: 270960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7714.416473] LNet: Removed LNI 192.168.206.117@tcp [ 7715.260955] Key type .llcrypt unregistered [ 7715.263824] Key type ._llcrypt unregistered [ 7742.351957] Key type ._llcrypt registered [ 7742.355694] Key type .llcrypt registered [ 7742.462910] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 7743.376636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7743.400073] alg: No test for adler32 (adler32-zlib) [ 7744.484430] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 7744.704600] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 7746.343168] Key type lgssc registered [ 7747.195494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7754.518223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7754.539738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7755.915127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7760.262709] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7765.480602] LustreError: 272376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7765.497261] LustreError: 272376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7769.005608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7769.512945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7774.718958] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7782.050732] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7787.637801] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7795.103070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7795.638541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7800.758856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7801.964933] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7810.216346] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7839.716182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7839.724392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7839.749336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7839.760994] Lustre: Skipped 1 previous similar message [ 7844.369210] Lustre: server umount lustre-OST0000 complete [ 7849.959802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7849.972938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7849.983819] Lustre: Skipped 1 previous similar message [ 7849.988661] Lustre: Skipped 1 previous similar message [ 7854.474691] Lustre: server umount lustre-MDT0000 complete [ 7855.072905] LustreError: 272376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7855.087551] LustreError: 272376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7859.048952] LustreError: 272357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837050 with bad export cookie 10791518268223108828 [ 7859.066255] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7859.069577] LustreError: 272357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7859.575793] Lustre: server umount lustre-MDT0001 complete [ 7873.676273] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 7876.265800] Key type lgssc unregistered [ 7876.695944] LNet: 276686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7876.707761] LNetError: 276686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7876.746077] LNet: Removed LNI 192.168.206.117@tcp [ 7877.842913] Key type .llcrypt unregistered [ 7877.847377] Key type ._llcrypt unregistered [ 7896.440184] Key type ._llcrypt registered [ 7896.442547] Key type .llcrypt registered [ 7896.599836] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 7897.561258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7897.649884] alg: No test for adler32 (adler32-zlib) [ 7898.896302] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 7899.238775] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 7900.991637] Key type lgssc registered [ 7902.368223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7910.351837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7910.368453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7911.942220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7916.821334] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7921.121719] LustreError: 277926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7921.147302] LustreError: 277926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7925.829757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7926.241790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7930.554097] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7937.220208] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7942.184724] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7949.477948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7950.068422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7956.227147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7956.912725] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7965.735907] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7976.935539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7976.935915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7976.948098] LustreError: Skipped 1 previous similar message [ 7976.949049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7976.969460] Lustre: Skipped 1 previous similar message [ 7982.056800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7982.062949] Lustre: Skipped 2 previous similar messages [ 7982.927114] Lustre: server umount lustre-OST0000 complete [ 7987.178993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7987.206462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7987.217689] Lustre: Skipped 1 previous similar message [ 7992.288738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7992.296045] Lustre: Skipped 1 previous similar message [ 7992.671270] Lustre: server umount lustre-MDT0000 complete [ 7995.982418] LustreError: 277891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837187 with bad export cookie 15877629558635273438 [ 7995.986407] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7995.997943] LustreError: 277891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7996.008624] LustreError: 277219:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 7996.301813] Lustre: server umount lustre-MDT0001 complete [ 8007.188177] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 8009.861621] Key type lgssc unregistered [ 8010.173744] LNet: 281631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8010.182596] LNetError: 281631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8010.196729] LNet: Removed LNI 192.168.206.117@tcp [ 8011.035527] Key type .llcrypt unregistered [ 8011.039176] Key type ._llcrypt unregistered [ 8021.811463] Key type ._llcrypt registered [ 8021.814721] Key type .llcrypt registered [ 8021.923022] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8038.227412] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 8055.725493] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 8057.256916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8057.275439] alg: No test for adler32 (adler32-zlib) [ 8058.626439] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 8058.929827] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 8060.591113] Key type lgssc registered [ 8061.690611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8092.491339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8092.545179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8093.800876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8093.829869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8093.922402] Lustre: lustre-MDT0000: new disk, initializing [ 8094.063479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8094.085553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8098.489097] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8107.642741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8107.765550] Lustre: 285182:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8107.858433] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8107.862189] Lustre: Skipped 1 previous similar message [ 8107.920987] Lustre: lustre-MDT0001: new disk, initializing [ 8108.048462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8108.084593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8108.097684] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8111.554663] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8121.241971] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8126.722934] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8133.677664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8133.958905] Lustre: lustre-OST0000: new disk, initializing [ 8133.964846] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8133.970869] Lustre: 287152:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8134.061886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8140.310801] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8144.478138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8144.485323] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8144.590057] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8150.163651] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8154.916967] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8155.113378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8159.536435] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8159.802919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8165.344802] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8165.356870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8165.363500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8165.387783] Lustre: Skipped 1 previous similar message [ 8167.927559] Lustre: server umount lustre-OST0000 complete [ 8174.573827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8174.585127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8174.597199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8174.602897] Lustre: Skipped 1 previous similar message [ 8175.593628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8175.618896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8175.628478] Lustre: Skipped 1 previous similar message [ 8180.707652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8180.722128] Lustre: Skipped 1 previous similar message [ 8185.827440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8185.834696] Lustre: Skipped 1 previous similar message [ 8186.336572] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8186.511179] Lustre: server umount lustre-MDT0000 complete [ 8189.688372] LustreError: 285173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837380 with bad export cookie 180503254905675138 [ 8189.696399] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8189.703821] LustreError: 285173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8190.105688] Lustre: server umount lustre-MDT0001 complete [ 8198.420934] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 09:29:48 (1787837388) [ 8204.366597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8204.810872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8208.860715] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8209.889654] LustreError: 289567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8209.901787] LustreError: 289567:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8213.993495] LustreError: 289568:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8217.130100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8217.415465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8221.345100] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8227.822953] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8233.187570] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8239.413215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8239.640222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8245.007982] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8252.180376] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8263.137199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8263.141758] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8263.161611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8267.457479] Lustre: server umount lustre-OST0000 complete [ 8273.890554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8273.899558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8273.922151] Lustre: Skipped 1 previous similar message [ 8276.967158] Lustre: server umount lustre-MDT0000 complete [ 8280.558554] LustreError: 290846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8280.579340] LustreError: 290846:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8280.650244] LustreError: 289548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837471 with bad export cookie 180503254905676013 [ 8280.656856] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8280.673740] LustreError: 289548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8281.099767] Lustre: server umount lustre-MDT0001 complete [ 8293.574586] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 8296.540989] Key type lgssc unregistered [ 8296.837685] LNet: 293105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8296.850027] LNetError: 293105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8296.873660] LNet: Removed LNI 192.168.206.117@tcp [ 8297.688926] Key type .llcrypt unregistered [ 8297.690783] Key type ._llcrypt unregistered [ 8322.343588] Key type ._llcrypt registered [ 8322.348877] Key type .llcrypt registered [ 8322.494712] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 8334.997659] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 8335.968930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8336.046308] alg: No test for adler32 (adler32-zlib) [ 8337.068782] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 8337.289225] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 8338.959137] Key type lgssc registered [ 8340.251411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8380.407587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8380.444025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8381.690408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8381.731340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8381.856706] Lustre: lustre-MDT0000: new disk, initializing [ 8381.935754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8381.949295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8385.837727] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8395.571438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8395.654914] Lustre: 296909:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8395.706918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8395.710570] Lustre: Skipped 1 previous similar message [ 8395.768938] Lustre: lustre-MDT0001: new disk, initializing [ 8395.839110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8395.871561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8395.881512] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8399.555363] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8407.956631] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8412.177922] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8418.624775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8419.020137] Lustre: lustre-OST2710: new disk, initializing [ 8419.029731] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8419.051959] Lustre: 298878:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8419.199294] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8425.032782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8425.041525] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8425.159394] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8425.398324] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8434.701737] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8442.006675] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8442.189681] Lustre: lustre-OST03e8: new disk, initializing [ 8442.193851] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8442.199206] Lustre: 299942:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8442.312208] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8447.827428] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8450.649863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8450.655692] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8450.789967] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8455.060916] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8462.430527] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8462.615097] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8467.028079] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8467.237580] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8471.403601] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8471.586403] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8475.877282] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8476.081445] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8481.764015] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8481.767217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8481.777350] Lustre: Skipped 3 previous similar messages [ 8481.786868] Lustre: Skipped 3 previous similar messages [ 8482.590186] Lustre: server umount lustre-MDT0000 complete [ 8485.664621] LustreError: 296901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837676 with bad export cookie 4051483267157807785 [ 8485.670866] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8485.675172] LustreError: 296901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8485.940411] Lustre: server umount lustre-MDT0001 complete [ 8499.644356] Lustre: server umount lustre-OST2710 complete [ 8502.241047] Lustre: 294709:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787837677/real 1787837677] req@000000009c8afe29 x1874683523825920/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787837693 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8502.259530] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8503.591410] Lustre: server umount lustre-OST03e8 complete [ 8520.035921] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 8528.137624] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 8567.672128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8567.860631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8567.890169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8567.972951] Lustre: lustre-MDT0000: new disk, initializing [ 8568.048496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8568.063731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8572.718426] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8582.786232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8582.864184] Lustre: 305191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8582.900604] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8582.904462] Lustre: Skipped 1 previous similar message [ 8582.963725] Lustre: lustre-MDT0001: new disk, initializing [ 8583.035865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8583.057761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8583.066718] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8587.671679] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8596.889752] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8601.875639] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8609.347219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8609.653569] Lustre: lustre-OST0000: new disk, initializing [ 8609.657936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8609.674438] Lustre: 307157:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8609.761996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8615.801613] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8618.554189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8618.563362] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8618.741275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8624.906249] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8629.598458] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8629.843529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8634.616744] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8634.817846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8639.461980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8639.465991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8639.493663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8639.506266] Lustre: Skipped 1 previous similar message [ 8642.639867] Lustre: server umount lustre-OST0000 complete [ 8649.696986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8649.700858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8649.718396] Lustre: Skipped 2 previous similar messages [ 8649.733500] Lustre: Skipped 1 previous similar message [ 8652.396793] Lustre: server umount lustre-MDT0000 complete [ 8656.523137] LustreError: 305182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837847 with bad export cookie 4051483267157809465 [ 8656.534768] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8656.890574] Lustre: server umount lustre-MDT0001 complete [ 8665.061683] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 09:37:35 (1787837855) [ 8666.849476] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8668.856616] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 09:37:38 (1787837858) [ 8672.581259] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 8696.673478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8696.919924] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8696.946390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8697.116841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8701.041234] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8708.992682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8709.117102] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8709.145908] Lustre: 311017:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8713.321367] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8734.185030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8734.441628] Lustre: Denying initial registration attempt from nid 192.168.206.117@tcp, specified as failover [ 8734.462978] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8734.472703] LustreError: 312470:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8734.483683] LustreError: 312470:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8734.489300] LustreError: 312470:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8734.590305] Lustre: server umount lustre-OST0000 complete [ 8734.593970] LustreError: 312470:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8745.444373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8745.446540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8745.457792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8745.480792] Lustre: Skipped 1 previous similar message [ 8747.443331] Lustre: server umount lustre-MDT0000 complete [ 8750.564491] LustreError: 311025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8750.582095] LustreError: 311025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8751.123270] LustreError: 311768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837942 with bad export cookie 4051483267157810347 [ 8751.127061] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8751.137106] LustreError: 311768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8751.433468] Lustre: server umount lustre-MDT0001 complete [ 8762.382057] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 8765.586910] Key type lgssc unregistered [ 8765.877284] LNet: 313594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8765.889463] LNetError: 313594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8765.902395] LNet: Removed LNI 192.168.206.117@tcp [ 8766.799344] Key type .llcrypt unregistered [ 8766.801053] Key type ._llcrypt unregistered [ 8777.682257] Key type ._llcrypt registered [ 8777.693291] Key type .llcrypt registered [ 8777.860941] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 8779.014478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8779.079704] alg: No test for adler32 (adler32-zlib) [ 8780.344651] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 8780.648634] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 8782.383334] Key type lgssc registered [ 8783.967358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8829.279718] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 8836.644475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8836.666658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8837.949477] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8837.968982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8838.107887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8841.395926] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8849.349343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8849.477751] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8849.501726] Lustre: 316360:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8849.742427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8853.356460] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8860.059909] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8865.490514] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8885.032648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8885.206316] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8885.450358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8890.946629] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8897.458886] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8903.141665] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8903.165920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8903.182960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8903.657777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8903.668859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8903.689239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8905.513492] Lustre: server umount lustre-OST0000 complete [ 8911.329218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8911.335797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8911.347542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8913.893038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8913.893985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8915.287154] Lustre: server umount lustre-MDT0000 complete [ 8918.666106] LustreError: 316353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838109 with bad export cookie 15632285945109809214 [ 8918.668219] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8918.680574] LustreError: 316353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8918.951665] Lustre: server umount lustre-MDT0001 complete [ 8929.450979] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 8932.032477] Key type lgssc unregistered [ 8932.293919] LNet: 319935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8932.302500] LNetError: 319935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8932.323873] LNet: Removed LNI 192.168.206.117@tcp [ 8933.218963] Key type .llcrypt unregistered [ 8933.227060] Key type ._llcrypt unregistered [ 8954.536169] Key type ._llcrypt registered [ 8954.539835] Key type .llcrypt registered [ 8954.668326] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 8955.479375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8955.614731] alg: No test for adler32 (adler32-zlib) [ 8956.629152] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 8956.828805] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 8958.468355] Key type lgssc registered [ 8959.629264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8966.744410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8966.767915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8968.180528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8972.946803] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8978.402339] LustreError: 321330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8978.436415] LustreError: 321330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8981.556023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8981.856254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8985.540418] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8993.871139] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9000.100616] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9009.097171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9009.445290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9016.165882] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9023.838120] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9037.047321] Lustre: server umount lustre-MDT0000 complete [ 9037.286534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9037.293242] LustreError: 323173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9037.296177] Lustre: Skipped 2 previous similar messages [ 9037.311566] LustreError: 323173:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9040.387652] LustreError: 321310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838231 with bad export cookie 18274347861517457046 [ 9040.398085] LustreError: 321310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9040.398585] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9041.104909] LustreError: 323173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9041.130642] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.17@tcp (stopping) [ 9042.401916] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9042.422912] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9045.408222] LustreError: 323630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9045.410655] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9045.458947] LustreError: 323630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9046.888445] Lustre: server umount lustre-MDT0001 complete [ 9054.996209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9064.463162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9064.928639] LustreError: 320641:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001754375a x1874684172861824/t0(0) o250->MGC192.168.206.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9065.273508] LustreError: 323163:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9065.306293] LustreError: 323163:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9065.375879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9065.455512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9069.826860] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9075.621475] LustreError: 324946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9075.623878] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9075.642747] LustreError: 324946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9078.515043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9079.026251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9084.182715] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9084.408857] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9091.776930] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9093.417468] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9099.408451] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9101.054050] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9106.400300] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9106.403803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9106.410778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9106.413190] Lustre: Skipped 1 previous similar message [ 9109.994951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9111.773545] Lustre: server umount lustre-OST0000 complete [ 9119.713100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9119.733078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9119.768594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9119.782847] Lustre: Skipped 2 previous similar messages [ 9122.195589] Lustre: server umount lustre-MDT0000 complete [ 9125.358051] LustreError: 324950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9125.387159] LustreError: 324950:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9126.870190] LustreError: 321310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838317 with bad export cookie 18274347861517464263 [ 9126.877161] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9126.887126] LustreError: 321310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9127.163321] Lustre: server umount lustre-MDT0001 complete [ 9138.783573] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 9141.559288] Key type lgssc unregistered [ 9141.752427] LNet: 327622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9141.763955] LNetError: 327622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9142.821865] LNet: Removed LNI 192.168.206.117@tcp [ 9143.877652] Key type .llcrypt unregistered [ 9143.882932] Key type ._llcrypt unregistered [ 9166.900543] Key type ._llcrypt registered [ 9166.905394] Key type .llcrypt registered [ 9166.988929] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 9167.828538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9167.964317] alg: No test for adler32 (adler32-zlib) [ 9169.074784] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 9169.333230] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 9171.039221] Key type lgssc registered [ 9172.340224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9179.999259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9180.016106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9181.442719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9185.928954] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9191.917471] LustreError: 329042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9191.941527] LustreError: 329042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9194.091269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9194.587712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9198.490186] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9204.896172] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9209.593360] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9212.902195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9212.918869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9212.922884] Lustre: Skipped 1 previous similar message [ 9218.020930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9218.031524] Lustre: Skipped 1 previous similar message [ 9218.356211] Lustre: server umount lustre-MDT0000 complete [ 9219.050165] LustreError: 329041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9219.074724] LustreError: 329041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9221.976200] LustreError: 329024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838413 with bad export cookie 3462333032060026131 [ 9221.980537] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9221.986793] LustreError: 329024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9222.406979] Lustre: server umount lustre-MDT0001 complete [ 9235.195604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9235.448089] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9235.489695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9235.697718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9240.200540] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9250.039984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9250.213648] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9250.241861] Lustre: 331629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9250.563585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9255.203353] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9263.478380] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9268.703853] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9276.646959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9276.816403] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9276.822881] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9276.828635] LustreError: 333511:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9276.833755] LustreError: 333511:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9276.839779] LustreError: 333511:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9276.922976] Lustre: server umount lustre-OST0000 complete [ 9276.927298] LustreError: 333511:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9286.059799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9286.270640] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9286.495109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9291.856809] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9295.472105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9298.536591] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9305.869778] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9305.955948] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9306.027174] Lustre: lustre-OST0001: new disk, initializing [ 9306.037059] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9306.047423] Lustre: 335049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9306.141056] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9312.044409] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9313.389468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9313.400742] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9313.492601] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9318.962550] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9324.003046] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9324.005968] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9324.016292] Lustre: Skipped 1 previous similar message [ 9324.018533] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9324.022385] LustreError: Skipped 1 previous similar message [ 9327.816581] Lustre: server umount lustre-OST0001 complete [ 9329.128425] LustreError: 333980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9329.138104] LustreError: 333980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9331.169958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9331.186515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9331.203018] Lustre: Skipped 1 previous similar message [ 9331.209175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9331.218660] Lustre: Skipped 1 previous similar message [ 9334.257492] LustreError: 333979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9334.272060] LustreError: 333979:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9337.297920] Lustre: server umount lustre-OST0000 complete [ 9342.945655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9342.956961] LustreError: Skipped 1 previous similar message [ 9342.959297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9342.968850] Lustre: Skipped 1 previous similar message [ 9342.973788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9342.977776] Lustre: Skipped 3 previous similar messages [ 9347.487234] Lustre: server umount lustre-MDT0000 complete [ 9349.600606] LustreError: 331634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9349.629783] LustreError: 331634:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9351.440744] LustreError: 331621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838542 with bad export cookie 3462333032060026691 [ 9351.441781] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9351.448327] LustreError: 331621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9351.880258] Lustre: server umount lustre-MDT0001 complete [ 9362.937224] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 9365.939605] Key type lgssc unregistered [ 9366.266696] LNet: 337012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9366.310753] LNetError: 337012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9366.330439] LNet: Removed LNI 192.168.206.117@tcp [ 9367.232695] Key type .llcrypt unregistered [ 9367.237142] Key type ._llcrypt unregistered [ 9437.813327] Key type ._llcrypt registered [ 9437.821220] Key type .llcrypt registered [ 9437.965926] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 9452.488748] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 9453.752859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9453.793480] alg: No test for adler32 (adler32-zlib) [ 9454.853411] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 9455.112019] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 9456.815171] Key type lgssc registered [ 9457.864894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9492.586668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9492.608235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9493.890866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9493.946458] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9494.023321] Lustre: lustre-MDT0000: new disk, initializing [ 9494.087821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9494.113648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9497.664907] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9508.342309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9508.537531] Lustre: 341947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9508.591412] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9508.598992] Lustre: Skipped 1 previous similar message [ 9508.798038] Lustre: lustre-MDT0001: new disk, initializing [ 9508.896268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9508.937311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9508.946739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9514.447355] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9526.526390] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9533.501470] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9541.021346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9541.380072] Lustre: lustre-OST0000: new disk, initializing [ 9541.387148] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9541.401594] Lustre: 343916:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9541.489459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9548.769698] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9549.945459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9549.960232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9550.160154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9559.552419] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9564.895382] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9565.140472] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9569.823273] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9570.033548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9575.913440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9575.914121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9575.916914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9575.928360] LustreError: Skipped 1 previous similar message [ 9575.953128] Lustre: Skipped 1 previous similar message [ 9581.033708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9581.044588] Lustre: Skipped 1 previous similar message [ 9586.143494] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9586.149822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9586.161866] Lustre: Skipped 1 previous similar message [ 9586.394868] Lustre: server umount lustre-OST0000 complete [ 9590.756632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9590.769290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9590.787223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9590.795552] Lustre: Skipped 1 previous similar message [ 9596.384877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9596.390134] Lustre: Skipped 3 previous similar messages [ 9597.007319] Lustre: server umount lustre-MDT0000 complete [ 9601.291173] LustreError: 343915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838792 with bad export cookie 10012346323871565552 [ 9601.296105] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9601.305910] LustreError: 343915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9601.699290] Lustre: server umount lustre-MDT0001 complete [ 9610.807189] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 09:53:21 (1787838801) [ 9617.680611] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 09:53:27 (1787838807) [ 9626.535099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9627.032100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9631.763940] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9632.225304] LustreError: 346672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9632.248268] LustreError: 346672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9637.355304] LustreError: 346673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9640.582087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9640.974862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9645.675544] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9653.333204] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9659.364885] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9665.464220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9665.835819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9671.975678] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9679.904707] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9688.274049] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9690.321370] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9691.976455] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9694.097159] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9696.325727] Lustre: Failing over lustre-MDT0000 [ 9696.720773] Lustre: server umount lustre-MDT0000 complete [ 9697.253919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9697.270601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9697.291044] Lustre: Skipped 1 previous similar message [ 9697.297125] LustreError: 346673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9702.883393] LustreError: 346672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9702.916192] LustreError: 346672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9708.005465] LustreError: 346672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9708.037099] LustreError: 346672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9713.141396] Lustre: 340168:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787838888/real 1787838888] req@00000000c5eec59d x1874684696140928/t0(0) o400->MGC192.168.206.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787838904 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9713.165866] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9715.310492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9718.239946] LustreError: 346673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9718.257658] LustreError: 346673:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 9724.384362] LustreError: 340167:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000039e254a1 x1874684696145792/t0(0) o250->MGC192.168.206.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9724.747838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9724.797304] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9728.751167] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9729.972947] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9730.036533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9730.075224] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 9730.144466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9739.564217] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9741.677499] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9745.161527] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9747.285933] Lustre: Failing over lustre-MDT0000 [ 9747.579523] Lustre: server umount lustre-MDT0000 complete [ 9749.246057] LustreError: 347677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9749.271343] LustreError: 347677:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 9750.496765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9750.502150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9750.517349] Lustre: Skipped 4 previous similar messages [ 9766.354855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9766.453503] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9766.696857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9766.770078] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9769.673239] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9771.101752] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9772.027498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9772.033550] Lustre: Skipped 2 previous similar messages [ 9772.078321] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9772.151078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9780.789059] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9782.478456] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9792.481558] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9792.496061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9792.500072] Lustre: Skipped 2 previous similar messages [ 9792.501184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9792.521780] Lustre: Skipped 2 previous similar messages [ 9794.818510] Lustre: server umount lustre-MDT0000 complete [ 9797.612235] LustreError: 346673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9797.635711] LustreError: 346673:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 9798.962875] LustreError: 346655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838990 with bad export cookie 10012346323871568261 [ 9798.972514] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9798.982310] LustreError: 346655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9799.365840] Lustre: server umount lustre-MDT0001 complete [ 9814.946279] Lustre: server umount lustre-OST0000 complete [ 9829.786737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9834.674638] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9843.628129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9843.944651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9843.953846] Lustre: Skipped 1 previous similar message [ 9848.293584] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9856.535439] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9862.334968] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9868.544170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9868.713721] LustreError: 354833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9868.737751] LustreError: 354833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9874.465098] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9875.437536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9881.707229] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9888.276876] Lustre: server umount lustre-OST0000 complete [ 9895.393386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9895.403688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9895.422098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9905.055150] Lustre: 340171:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787839080/real 1787839080] req@0000000034706541 x1874684696210176/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787839096 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9906.079218] Lustre: 340168:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787839081/real 1787839081] req@0000000039023b80 x1874684696210944/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787839097 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9906.143476] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9906.329533] Lustre: server umount lustre-MDT0000 complete [ 9909.484786] LustreError: 352993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839100 with bad export cookie 10012346323871569003 [ 9909.496505] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9909.497328] LustreError: 352993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9909.802238] Lustre: server umount lustre-MDT0001 complete [ 9920.341825] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 9922.744702] Key type lgssc unregistered [ 9923.008814] LNet: 356547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9923.013518] LNetError: 356547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9923.034132] LNet: Removed LNI 192.168.206.117@tcp [ 9923.803430] Key type .llcrypt unregistered [ 9923.808191] Key type ._llcrypt unregistered [ 9948.546176] Key type ._llcrypt registered [ 9948.552387] Key type .llcrypt registered [ 9948.688525] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 9949.560618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9949.648316] alg: No test for adler32 (adler32-zlib) [ 9950.722419] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 9950.995152] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 9952.657523] Key type lgssc registered [ 9953.692772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9960.233615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9960.253843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9961.671425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9965.675381] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9971.176914] LustreError: 358046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9971.194941] LustreError: 358046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9973.200728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9973.515598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9977.478615] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9983.172878] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9987.571556] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9993.465807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9993.929918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9999.045912] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10005.925434] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10006.267404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10014.691931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10014.698854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10014.717917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10016.737581] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10016.737675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10016.749376] Lustre: Skipped 1 previous similar message [10016.756060] Lustre: Skipped 2 previous similar messages [10020.933852] Lustre: server umount lustre-MDT0000 complete [10021.858295] LustreError: 358773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10021.884660] LustreError: 358773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10024.335573] LustreError: 358026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839215 with bad export cookie 3952709102136891635 [10024.345035] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10024.351212] LustreError: 358026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10024.584738] Lustre: server umount lustre-MDT0001 complete [10038.218302] Lustre: server umount lustre-OST0000 complete [10051.844918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10052.353164] LustreError: 361757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10052.454634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10056.804519] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10057.698247] LustreError: 361758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10062.820535] LustreError: 361757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10064.371215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10064.646135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10068.154159] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10074.750352] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10079.172641] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10084.952608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10085.192372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10090.009568] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10096.606079] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10097.518513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10116.071696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10116.082605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10116.105749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10118.112666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10118.114103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10118.132626] Lustre: Skipped 1 previous similar message [10118.146548] Lustre: Skipped 1 previous similar message [10119.445732] Lustre: server umount lustre-MDT0000 complete [10123.044892] LustreError: 361736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839314 with bad export cookie 3952709102136892748 [10123.048480] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10123.052933] LustreError: 361736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10123.281635] Lustre: server umount lustre-MDT0001 complete [10137.356420] Lustre: server umount lustre-OST0000 complete [10153.442808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10153.720374] LustreError: 365680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10153.791587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10157.207617] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10164.976374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10169.576099] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10178.081815] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10183.960401] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10190.849984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10191.121262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10191.130619] Lustre: Skipped 1 previous similar message [10197.206197] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10198.336924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10204.948162] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10210.465482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10210.468090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10210.495887] Lustre: Skipped 2 previous similar messages [10211.696550] Lustre: server umount lustre-OST0000 complete [10221.034414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10221.040862] Lustre: Skipped 3 previous similar messages [10221.824709] Lustre: server umount lustre-MDT0000 complete [10225.703905] LustreError: 365662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839416 with bad export cookie 3952709102136893763 [10225.717193] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10225.729626] LustreError: 365662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10226.113683] Lustre: server umount lustre-MDT0001 complete [10236.713992] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [10239.235959] Key type lgssc unregistered [10239.542577] LNet: 369215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10239.549348] LNetError: 369215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10239.571238] LNet: Removed LNI 192.168.206.117@tcp [10240.290504] Key type .llcrypt unregistered [10240.292454] Key type ._llcrypt unregistered [10264.062644] Key type ._llcrypt registered [10264.064640] Key type .llcrypt registered [10264.136062] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [10264.900134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10264.951823] alg: No test for adler32 (adler32-zlib) [10266.002692] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [10266.265983] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [10267.985078] Key type lgssc registered [10268.873482] Lustre: Echo OBD driver; http://www.lustre.org/ [10274.932198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10274.943438] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10274.948551] LustreError: 370656:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10274.959956] LustreError: 370656:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10274.964201] LustreError: 370656:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10274.970899] LustreError: 370656:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10274.975590] LustreError: 370656:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10282.364699] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10282.376458] LustreError: 371053:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10282.405852] LustreError: 371053:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10282.415945] LustreError: 371053:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10282.423864] LustreError: 371053:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10282.431750] LustreError: 371053:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10302.971644] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [10305.239903] Key type lgssc unregistered [10305.490350] LNet: 372060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10305.499967] LNetError: 372060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10305.516620] LNet: Removed LNI 192.168.206.117@tcp [10306.212423] Key type .llcrypt unregistered [10306.214399] Key type ._llcrypt unregistered [10322.331699] Key type ._llcrypt registered [10322.335464] Key type .llcrypt registered [10322.462732] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [10335.619220] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [10336.280155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10336.297527] alg: No test for adler32 (adler32-zlib) [10337.250277] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [10337.468630] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [10339.111148] Key type lgssc registered [10339.905546] Lustre: Echo OBD driver; http://www.lustre.org/ [10373.469629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10373.507461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10374.819792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10374.885491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10374.979215] Lustre: lustre-MDT0000: new disk, initializing [10375.072084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10375.082649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10380.095313] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10391.270945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10391.466838] Lustre: 375217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10391.536850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10391.540637] Lustre: Skipped 1 previous similar message [10391.628748] Lustre: lustre-MDT0001: new disk, initializing [10391.733141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10391.767083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10391.782723] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10396.555622] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10407.435725] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10414.003204] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10421.855757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10422.173283] Lustre: lustre-OST0000: new disk, initializing [10422.184864] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10422.193232] Lustre: 377183:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10422.328779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10428.371815] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10428.981101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10428.987337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10429.068302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10437.431303] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10441.711953] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10441.962766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10446.503180] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10446.693326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10449.389407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10449.402505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10449.412565] Lustre: Skipped 1 previous similar message [10454.501291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10454.508261] Lustre: Skipped 1 previous similar message [10454.736658] Lustre: server umount lustre-OST0000 complete [10458.595438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10458.603107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10458.616130] Lustre: Skipped 1 previous similar message [10458.619856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10459.620216] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10464.484146] Lustre: server umount lustre-MDT0000 complete [10467.908015] LustreError: 377182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839659 with bad export cookie 14109339957418348832 [10467.915990] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10467.930318] LustreError: 377182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10468.275967] Lustre: server umount lustre-MDT0001 complete [10476.206733] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10477.602151] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 10:07:47 (1787839667) [10483.284564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10483.530910] LustreError: 379646:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10483.544644] LustreError: 379646:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10483.582684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10487.602211] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10488.806910] LustreError: 379647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10493.924689] LustreError: 379646:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10496.257274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10496.659227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10501.039379] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10508.918574] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10513.924671] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10519.384271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10519.661491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10525.653232] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10532.152703] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10538.308856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10538.467741] Lustre: lustre-OST0001: new disk, initializing [10538.473519] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10538.484230] Lustre: 382482:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10538.557200] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10543.947320] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10548.326755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10548.334898] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10548.385467] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10552.238606] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10558.949231] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10558.953908] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10558.961719] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10558.961726] Lustre: Skipped 2 previous similar messages [10558.970137] Lustre: Skipped 1 previous similar message [10558.993816] LustreError: Skipped 1 previous similar message [10561.530865] Lustre: server umount lustre-OST0001 complete [10564.073914] LustreError: 381467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10564.097629] LustreError: 381467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10569.207124] LustreError: 381469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10569.219712] LustreError: 381469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10578.400314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10578.415584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10578.434943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10578.441457] Lustre: Skipped 2 previous similar messages [10579.429697] LustreError: 381467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10579.446070] LustreError: 381467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10583.300195] Lustre: server umount lustre-OST0000 complete [10588.641737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10588.648085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10588.657460] Lustre: Skipped 1 previous similar message [10588.662427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10588.667577] Lustre: Skipped 2 previous similar messages [10592.593093] Lustre: server umount lustre-MDT0000 complete [10596.196704] LustreError: 379627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839787 with bad export cookie 14109339957418349714 [10596.206053] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10596.211527] LustreError: 379627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10602.680444] Lustre: server umount lustre-MDT0001 complete [10613.189040] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [10615.714941] Key type lgssc unregistered [10616.024354] LNet: 384475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10616.035469] LNetError: 384475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10616.059390] LNet: Removed LNI 192.168.206.117@tcp [10616.953039] Key type .llcrypt unregistered [10616.955253] Key type ._llcrypt unregistered [10666.831600] Key type ._llcrypt registered [10666.834570] Key type .llcrypt registered [10666.952682] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [10667.819126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10667.868299] alg: No test for adler32 (adler32-zlib) [10668.928772] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [10669.336282] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [10671.088156] Key type lgssc registered [10672.209169] Lustre: Echo OBD driver; http://www.lustre.org/ [10679.383110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10679.397399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10680.633468] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10680.659699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10680.759756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10684.862565] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10693.700673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10693.895046] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10693.943984] Lustre: 387272:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10694.297245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10699.403868] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10707.522642] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10714.159658] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10719.725431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10719.739115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10719.760636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10721.250550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10721.252283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10721.277047] Lustre: Skipped 1 previous similar message [10722.691911] Lustre: server umount lustre-MDT0000 complete [10725.561429] LustreError: 387263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839916 with bad export cookie 5971772830426088380 [10725.564933] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10725.571955] LustreError: 387263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10725.939626] Lustre: server umount lustre-MDT0001 complete [10734.054240] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 10:12:04 (1787839924) [10739.174546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10739.503213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10743.013898] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10744.800668] LustreError: 389784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10744.825659] LustreError: 389784:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10749.927212] LustreError: 389783:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10750.402640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10750.785932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10754.421743] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10762.288443] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10767.772813] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10773.963951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10774.164723] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10774.457538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10780.330550] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10785.595520] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10792.128748] Lustre: Permanently deactivating lustre-OST0000 [10792.144567] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10798.095195] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10799.530023] LustreError: 392772:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10799.536245] LustreError: 392772:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10799.539616] LustreError: 392772:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10801.088465] LustreError: 392820:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10801.092884] LustreError: 392820:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10808.549363] Lustre: server umount lustre-OST0000 complete [10810.392511] LustreError: 393069:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10810.398481] LustreError: 393069:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10812.386240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10812.394792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10812.404208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10815.970710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10815.973657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10815.984871] Lustre: Skipped 1 previous similar message [10821.090413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10821.096697] Lustre: Skipped 1 previous similar message [10826.208653] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10826.381937] Lustre: server umount lustre-MDT0000 complete [10829.125817] LustreError: 390518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840020 with bad export cookie 5971772830426088961 [10829.131554] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10829.138871] LustreError: 390518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10839.680134] Lustre: server umount lustre-MDT0001 complete [10847.227178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10851.502682] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10854.611170] LustreError: 394229:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10861.194378] Lustre: 394568:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.117@tcp [10863.449331] Lustre: server umount MGS complete [10875.095986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10875.530793] LustreError: 395292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10875.549160] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10875.557236] Lustre: Skipped 1 previous similar message [10875.624985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10879.532290] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10881.010896] LustreError: 395293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10886.121664] LustreError: 395292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10888.090126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10888.302317] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10892.316338] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10899.247225] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10903.373197] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10904.742516] Lustre: Permanently reactivating lustre-OST0000 [10904.757272] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10904.764955] Lustre: Skipped 2 previous similar messages [10931.167147] LustreError: 397042:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10937.573644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10937.878539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10937.884941] Lustre: Skipped 1 previous similar message [10942.431723] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10948.970616] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10955.233734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10955.256047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10955.268642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10955.272908] Lustre: Skipped 2 previous similar messages [10960.276421] Lustre: server umount lustre-OST0000 complete [10965.472527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10965.483356] LustreError: Skipped 1 previous similar message [10965.491863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10965.511207] Lustre: Skipped 1 previous similar message [10969.973520] Lustre: server umount lustre-MDT0000 complete [10971.619578] LustreError: 396364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10971.651776] LustreError: 396364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10973.700302] LustreError: 395273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840164 with bad export cookie 5971772830426090011 [10973.708134] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10973.713478] LustreError: 395273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10974.022808] Lustre: server umount lustre-MDT0001 complete [10982.913482] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [10985.179957] Key type lgssc unregistered [10985.446604] LNet: 399707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10985.454614] LNetError: 399707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10985.479593] LNet: Removed LNI 192.168.206.117@tcp [10986.385166] Key type .llcrypt unregistered [10986.391705] Key type ._llcrypt unregistered [11004.075801] Key type ._llcrypt registered [11004.079229] Key type .llcrypt registered [11004.255809] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [11023.450891] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [11024.804777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11024.867229] alg: No test for adler32 (adler32-zlib) [11025.899701] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [11026.137303] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [11027.855234] Key type lgssc registered [11028.966680] Lustre: Echo OBD driver; http://www.lustre.org/ [11062.063880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11062.100945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11063.387301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11063.415183] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11063.478179] Lustre: lustre-MDT0000: new disk, initializing [11063.538702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11063.554340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11067.796904] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11076.934298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11077.042838] Lustre: 402863:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11077.100617] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11077.107038] Lustre: Skipped 1 previous similar message [11077.184826] Lustre: lustre-MDT0001: new disk, initializing [11077.266638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11077.306954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11077.321158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11082.344586] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11090.683880] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11095.764347] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11101.783051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11101.980425] Lustre: lustre-OST0000: new disk, initializing [11101.984341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11101.990650] Lustre: 404830:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11102.063272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11106.844654] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11111.518515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11111.529698] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11111.638769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11117.521292] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11123.046599] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11123.331654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11128.374259] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11128.567751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11132.384542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11132.393915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11132.396458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11132.407489] Lustre: Skipped 1 previous similar message [11136.566987] Lustre: server umount lustre-OST0000 complete [11142.630222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11142.641413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11142.648653] Lustre: Skipped 1 previous similar message [11142.651014] Lustre: Skipped 1 previous similar message [11145.906355] Lustre: server umount lustre-MDT0000 complete [11149.393388] LustreError: 403658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840340 with bad export cookie 9755253147662195689 [11149.401900] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11149.411382] LustreError: 403658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11149.812457] Lustre: server umount lustre-MDT0001 complete [11157.834433] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11159.728570] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 10:19:09 (1787840349) [11166.451941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11166.865879] LustreError: 407290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11166.884927] LustreError: 407290:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11166.955974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11171.478465] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11172.321168] LustreError: 407291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11176.418262] LustreError: 407290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11179.794862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11180.305847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11185.559354] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11193.577569] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11199.550390] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11206.369168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11206.753328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11213.912236] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11222.375887] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11223.987405] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11226.334230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11232.739058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11232.745501] LustreError: Skipped 1 previous similar message [11232.750681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11232.769874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11232.779244] Lustre: Skipped 1 previous similar message [11235.814600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11235.820217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11235.827065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11235.849964] Lustre: Skipped 1 previous similar message [11237.670585] Lustre: server umount lustre-OST0000 complete [11241.953720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11241.960190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11241.968113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11247.405539] Lustre: server umount lustre-MDT0000 complete [11250.305693] LustreError: 410264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840441 with bad export cookie 9755253147662196571 [11250.313767] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11250.314104] LustreError: 410264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11250.615029] Lustre: server umount lustre-MDT0001 complete [11260.151261] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [11262.766152] Key type lgssc unregistered [11263.019385] LNet: 411019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11263.025966] LNetError: 411019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11263.043240] LNet: Removed LNI 192.168.206.117@tcp [11263.807148] Key type .llcrypt unregistered [11263.809348] Key type ._llcrypt unregistered [11283.504682] Key type ._llcrypt registered [11283.506703] Key type .llcrypt registered [11283.593397] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [11283.851467] Key type .llcrypt unregistered [11283.857696] Key type ._llcrypt unregistered [11300.198031] Key type ._llcrypt registered [11300.201646] Key type .llcrypt registered [11300.352211] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [11301.148133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11301.196152] alg: No test for adler32 (adler32-zlib) [11302.379281] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [11302.545643] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [11304.231825] Key type lgssc registered [11305.414524] Lustre: Echo OBD driver; http://www.lustre.org/ [11313.851300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11313.899332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11315.549438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11319.861104] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11325.927126] LustreError: 413241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11325.971037] LustreError: 413241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11327.786725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11328.062638] LustreError: 413242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11328.111304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11333.097733] LustreError: 413237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11333.114562] LustreError: 413237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11333.657449] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11338.209293] LustreError: 413241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11338.234183] LustreError: 413241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11340.864415] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11341.291897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11346.033484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11346.306363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11351.175310] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11361.760800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11361.766575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11361.769259] LustreError: Skipped 1 previous similar message [11361.772633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11361.777947] Lustre: Skipped 1 previous similar message [11366.888366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11366.899454] Lustre: Skipped 2 previous similar messages [11367.681486] Lustre: server umount lustre-OST0000 complete [11372.002565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11372.013397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11372.014947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11372.032800] Lustre: Skipped 1 previous similar message [11377.126323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11377.135479] Lustre: Skipped 2 previous similar messages [11378.240447] Lustre: server umount lustre-MDT0000 complete [11381.475183] LustreError: 413224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840572 with bad export cookie 11908546814390486414 [11381.478391] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11381.485444] LustreError: 413224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11381.704145] Lustre: server umount lustre-MDT0001 complete [11391.523096] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [11394.024445] Key type lgssc unregistered [11394.365325] LNet: 416392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11394.371183] LNetError: 416392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11395.429680] LNet: Removed LNI 192.168.206.117@tcp [11396.280916] Key type .llcrypt unregistered [11396.286320] Key type ._llcrypt unregistered [11416.795430] Key type ._llcrypt registered [11416.797897] Key type .llcrypt registered [11416.892812] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [11417.743735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11417.766664] alg: No test for adler32 (adler32-zlib) [11418.796595] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [11418.983912] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [11420.634119] Key type lgssc registered [11421.594450] Lustre: Echo OBD driver; http://www.lustre.org/ [11427.930585] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11427.968700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11443.551484] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11448.671409] LustreError: 417763:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.117@tcp: failed processing log, type 4: rc = -110 [11475.296294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11480.279605] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11486.030467] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11491.467372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11491.834204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11491.895022] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11495.198821] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11496.932370] LustreError: 417789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11496.948117] LustreError: 417789:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11501.589430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11501.856206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11505.558332] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11511.257910] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11516.066611] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11522.528858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11522.534738] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11522.536486] LustreError: Skipped 1 previous similar message [11522.537728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11522.548240] Lustre: Skipped 1 previous similar message [11522.561679] Lustre: Skipped 1 previous similar message [11525.602969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11527.988095] Lustre: server umount lustre-OST0000 complete [11532.769762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11532.775252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11532.784656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11532.793786] Lustre: Skipped 1 previous similar message [11535.844534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11535.846834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11535.879172] Lustre: Skipped 1 previous similar message [11537.867585] Lustre: server umount lustre-MDT0000 complete [11540.961116] LustreError: 420048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11541.001679] LustreError: 420048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11543.524084] LustreError: 417769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840734 with bad export cookie 17763712140139934561 [11543.530492] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11543.539982] LustreError: 417769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11544.348957] Lustre: server umount lustre-MDT0001 complete [11558.205166] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [11560.776878] Key type lgssc unregistered [11561.063377] LNet: 421338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11561.078751] LNetError: 421338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11561.105403] LNet: Removed LNI 192.168.206.117@tcp [11562.042482] Key type .llcrypt unregistered [11562.046085] Key type ._llcrypt unregistered [11584.887240] Key type ._llcrypt registered [11584.891470] Key type .llcrypt registered [11584.998164] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [11585.684936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11585.707409] alg: No test for adler32 (adler32-zlib) [11586.759968] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [11586.960659] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [11588.623180] Key type lgssc registered [11590.012231] Lustre: Echo OBD driver; http://www.lustre.org/ [11597.559462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11597.576497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11598.887756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11601.987498] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11608.034083] LustreError: 422745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11608.052147] LustreError: 422745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11610.418824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11610.653876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11614.214655] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11620.409973] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11626.857221] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11632.391153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11632.615178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11637.242893] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11642.982571] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11644.910667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11651.552393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11651.558771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11651.572161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11654.884594] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.17@tcp (stopping) [11655.141158] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11655.156758] Lustre: Skipped 1 previous similar message [11657.692506] Lustre: server umount lustre-MDT0000 complete [11659.907323] LustreError: 422726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.17@tcp arrived at 1787840850 with bad export cookie 13007403646178680405 [11659.916936] LustreError: 422726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11660.257374] LustreError: 422746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11660.278532] LustreError: 422746:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11661.832811] LustreError: 422728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840852 with bad export cookie 13007403646178679740 [11661.847083] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11665.377538] LustreError: 422745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11665.379163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11665.406711] LustreError: 422745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11665.443751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11665.451656] Lustre: Skipped 3 previous similar messages [11668.259102] Lustre: server umount lustre-OST0000 complete [11671.525720] LustreError: 422745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11671.560237] LustreError: 422745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11675.653923] Lustre: server umount lustre-MDT0001 complete [11685.147932] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [11687.763223] Key type lgssc unregistered [11688.062709] LNet: 426431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11688.070645] LNetError: 426431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11688.115180] LNet: Removed LNI 192.168.206.117@tcp [11688.832759] Key type .llcrypt unregistered [11688.837174] Key type ._llcrypt unregistered [11708.098326] Key type ._llcrypt registered [11708.102725] Key type .llcrypt registered [11708.240710] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [11709.004338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11709.017474] alg: No test for adler32 (adler32-zlib) [11710.080620] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [11710.319910] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [11711.994052] Key type lgssc registered [11712.862731] Lustre: Echo OBD driver; http://www.lustre.org/ [11720.045819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11720.058190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11721.579769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11726.765442] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11730.914989] LustreError: 427843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11730.924821] LustreError: 427843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11734.107863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11734.322458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11738.097855] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11744.923601] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11749.537095] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11755.544876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11755.887326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11760.985972] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11766.121419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11767.438437] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11775.199202] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.17@tcp (stopping) [11775.459649] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11775.469867] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11776.485780] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11776.489251] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11776.490219] Lustre: Skipped 1 previous similar message [11778.544326] Lustre: server umount lustre-MDT0001 complete [11780.319897] LustreError: 427840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11781.602470] LustreError: 427844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11781.631593] LustreError: 427844:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11786.721137] LustreError: 427843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11786.722411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11786.726686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11786.730798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11786.730806] Lustre: Skipped 1 previous similar message [11790.936610] Lustre: server umount lustre-OST0000 complete [11791.857400] LustreError: 427844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11796.796331] Lustre: server umount lustre-MDT0000 complete [11810.830711] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [11813.260598] Key type lgssc unregistered [11813.533928] LNet: 431532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11813.544158] LNetError: 431532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11813.563167] LNet: Removed LNI 192.168.206.117@tcp [11814.336558] Key type .llcrypt unregistered [11814.346896] Key type ._llcrypt unregistered [11835.428106] Key type ._llcrypt registered [11835.430428] Key type .llcrypt registered [11835.550273] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [11849.573409] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [11850.556392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11850.565358] alg: No test for adler32 (adler32-zlib) [11851.697914] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [11852.013666] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [11853.657200] Key type lgssc registered [11854.810726] Lustre: Echo OBD driver; http://www.lustre.org/ [11889.819116] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [11906.753023] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11906.788092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11908.073806] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11908.089982] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11908.198879] Lustre: lustre-MDT0000: new disk, initializing [11908.287585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11908.309111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11913.854334] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11929.512643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11929.696922] Lustre: lustre-OST0000: new disk, initializing [11929.701486] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11929.706202] Lustre: Skipped 1 previous similar message [11929.711051] Lustre: 436579:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11929.782991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11935.660522] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11935.821942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11935.830520] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11935.856416] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11945.475347] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11959.707328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11959.900760] Lustre: 435560:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11959.989391] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11960.096988] Lustre: lustre-MDT0001: new disk, initializing [11960.278885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11960.347403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11960.361875] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11964.533872] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11973.600319] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11973.828774] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11978.568037] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11978.824237] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11983.847769] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11983.849994] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11983.875149] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11983.893030] Lustre: Skipped 1 previous similar message [11988.188619] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.17@tcp (stopping) [11990.046573] Lustre: server umount lustre-MDT0001 complete [11999.222091] LustreError: 435573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11999.254822] LustreError: 435573:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12004.332095] LustreError: 435572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12004.360701] LustreError: 435572:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12009.441490] LustreError: 435573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12009.448683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12009.498048] Lustre: Skipped 1 previous similar message [12009.519884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12009.525886] Lustre: Skipped 2 previous similar messages [12014.571917] LustreError: 435572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12014.573583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12019.685585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12019.688379] LustreError: 435573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12023.263129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12023.416566] Lustre: server umount lustre-OST0000 complete [12041.187074] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12041.595792] Lustre: server umount lustre-MDT0000 complete [12055.413568] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [12058.135557] Key type lgssc unregistered [12058.405368] LNet: 440328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12058.412819] LNetError: 440328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12058.441375] LNet: Removed LNI 192.168.206.117@tcp [12059.449165] Key type .llcrypt unregistered [12059.450942] Key type ._llcrypt unregistered [12100.414733] Key type ._llcrypt registered [12100.417898] Key type .llcrypt registered [12100.506511] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [12100.861268] Key type .llcrypt unregistered [12100.862946] Key type ._llcrypt unregistered [12118.574558] Key type ._llcrypt registered [12118.576924] Key type .llcrypt registered [12118.685613] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [12119.821040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12119.838832] alg: No test for adler32 (adler32-zlib) [12121.256841] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [12121.615798] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [12123.455602] Key type lgssc registered [12124.982826] Lustre: Echo OBD driver; http://www.lustre.org/ [12156.457122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12156.520579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12157.764117] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12157.792942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12157.853764] Lustre: lustre-MDT0000: new disk, initializing [12157.899323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12157.929947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12162.107774] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [12172.341580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12172.487382] Lustre: 443715:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12172.556967] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12172.563869] Lustre: Skipped 1 previous similar message [12172.671710] Lustre: lustre-MDT0001: new disk, initializing [12172.785973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12172.814924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12172.822771] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12176.595518] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [12184.974602] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12189.938613] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12195.292117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12195.547143] Lustre: lustre-OST0000: new disk, initializing [12195.550957] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12195.562018] Lustre: 445684:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12195.650222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12201.088892] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [12201.638396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12201.650159] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12201.764292] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12209.557271] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12217.317052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12217.325971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12217.340877] Lustre: Skipped 2 previous similar messages [12217.358500] Lustre: Skipped 2 previous similar messages [12221.321639] Lustre: server umount lustre-MDT0000 complete [12225.220933] LustreError: 445682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841416 with bad export cookie 14384103136652183696 [12225.227765] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12225.234531] LustreError: 445682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12225.484441] Lustre: server umount lustre-MDT0001 complete [12239.620995] Lustre: server umount lustre-OST0000 complete [12249.765434] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 10:37:19 (1787841439) [12257.493781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12257.904470] LustreError: 447711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12257.926874] LustreError: 447711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12257.994247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12262.376531] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [12263.393277] LustreError: 447712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12268.515191] LustreError: 447711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12270.533478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12270.819351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12275.337212] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [12282.221415] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12287.784853] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12307.395533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12307.647606] Lustre: Found index 0 for lustre-OST0000, updating log [12307.844767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12313.850740] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [12322.680711] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12329.338918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12340.706675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12340.720807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12340.750804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12341.794875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12341.803125] Lustre: Skipped 2 previous similar messages [12346.235285] Lustre: server umount lustre-OST0000 complete [12363.234826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12363.236746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12363.246183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12363.261072] Lustre: Skipped 2 previous similar messages [12367.393670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12367.402641] Lustre: Skipped 2 previous similar messages [12368.963989] Lustre: server umount lustre-MDT0000 complete [12371.551217] LNet: 1 local NIs in recovery (showing 1): 192.168.206.117@tcp [12373.123641] LustreError: 449551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841564 with bad export cookie 14384103136652185096 [12373.126791] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12373.137177] LustreError: 449551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12373.510652] Lustre: server umount lustre-MDT0001 complete [12382.672784] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 10:39:32 (1787841572) [12999.829083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13000.307929] LustreError: 454143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13000.387580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13004.929396] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [13005.793177] LustreError: 454144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13010.917934] LustreError: 454143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13012.460491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13012.763030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13017.775400] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [13026.030400] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13031.769770] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13051.048870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13051.214949] Lustre: Found index 0 for lustre-OST0000, updating log [13051.406848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13056.667434] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [13062.751444] LNet: 1 local NIs in recovery (showing 1): 192.168.206.117@tcp [13063.742663] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13065.773860] Lustre: server umount lustre-OST0000 complete [13070.500896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13070.503188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13070.519449] Lustre: Skipped 1 previous similar message [13070.524289] Lustre: Skipped 2 previous similar messages [13071.597717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13071.601361] Lustre: Skipped 2 previous similar messages [13073.635300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13073.640668] Lustre: Skipped 3 previous similar messages [13075.910290] Lustre: server umount lustre-MDT0000 complete [13076.704614] LustreError: 454872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13076.727296] LustreError: 454872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13079.435910] LustreError: 454123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842270 with bad export cookie 14384103136652186181 [13079.443285] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13079.450854] LustreError: 454123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13079.740401] Lustre: server umount lustre-MDT0001 complete [13108.346054] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [13117.183514] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [13153.105212] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 10:52:23 (1787842343) [13168.755820] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [13171.241251] Key type lgssc unregistered [13171.566858] LNet: 460756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13171.576983] LNetError: 460756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13260.711762] LNet: Removed LNI 192.168.206.117@tcp [13261.618045] Key type .llcrypt unregistered [13261.619773] Key type ._llcrypt unregistered [13277.551454] Key type ._llcrypt registered [13277.554029] Key type .llcrypt registered [13277.650529] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [13278.200188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13278.228105] alg: No test for adler32 (adler32-zlib) [13279.300332] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [13279.564853] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [13281.271392] Key type lgssc registered [13282.415107] Lustre: Echo OBD driver; http://www.lustre.org/ [13286.646374] LNet: Added LNI 192.168.206.117@tcp100 [8/256/0/180] [13287.830139] LNet: Added LNI 192.168.206.117@tcp101 [8/256/0/180] [13289.304544] LNet: Added LNI 192.168.206.117@tcp102 [8/256/0/180] [13290.403567] LNet: Added LNI 192.168.206.117@tcp103 [8/256/0/180] [13291.815880] LNet: Added LNI 192.168.206.117@tcp104 [8/256/0/180] [13293.286496] LNet: Added LNI 192.168.206.117@tcp105 [8/256/0/180] [13294.142679] LNet: Added LNI 192.168.206.117@tcp106 [8/256/0/180] [13295.010670] LNet: Added LNI 192.168.206.117@tcp107 [8/256/0/180] [13295.815688] LNet: Added LNI 192.168.206.117@tcp108 [8/256/0/180] [13297.162185] LNet: Added LNI 192.168.206.117@tcp109 [8/256/0/180] [13298.556765] LNet: Added LNI 192.168.206.117@tcp110 [8/256/0/180] [13299.693949] LNet: Added LNI 192.168.206.117@tcp111 [8/256/0/180] [13300.817979] LNet: Added LNI 192.168.206.117@tcp112 [8/256/0/180] [13301.871716] LNet: Added LNI 192.168.206.117@tcp113 [8/256/0/180] [13302.687151] LNet: Added LNI 192.168.206.117@tcp114 [8/256/0/180] [13303.837501] LNet: Added LNI 192.168.206.117@tcp115 [8/256/0/180] [13305.206380] LNet: Added LNI 192.168.206.117@tcp116 [8/256/0/180] [13306.355332] LNet: Added LNI 192.168.206.117@tcp117 [8/256/0/180] [13307.739490] LNet: Added LNI 192.168.206.117@tcp118 [8/256/0/180] [13308.750172] LNet: Added LNI 192.168.206.117@tcp119 [8/256/0/180] [13309.602594] LNet: Added LNI 192.168.206.117@tcp120 [8/256/0/180] [13310.400783] LNet: Added LNI 192.168.206.117@tcp121 [8/256/0/180] [13311.523541] LNet: Added LNI 192.168.206.117@tcp122 [8/256/0/180] [13312.366758] LNet: Added LNI 192.168.206.117@tcp123 [8/256/0/180] [13313.061990] LNet: Added LNI 192.168.206.117@tcp124 [8/256/0/180] [13313.880324] LNet: Added LNI 192.168.206.117@tcp125 [8/256/0/180] [13314.984200] LNet: Added LNI 192.168.206.117@tcp126 [8/256/0/180] [13315.943823] LNet: Added LNI 192.168.206.117@tcp127 [8/256/0/180] [13317.275338] LNet: Added LNI 192.168.206.117@tcp128 [8/256/0/180] [13318.361296] LNet: Added LNI 192.168.206.117@tcp129 [8/256/0/180] [13319.684334] LNet: Added LNI 192.168.206.117@tcp130 [8/256/0/180] [13320.715742] LNet: Added LNI 192.168.206.117@tcp131 [8/256/0/180] [13321.563313] LNet: Added LNI 192.168.206.117@tcp132 [8/256/0/180] [13322.368933] LNet: Added LNI 192.168.206.117@tcp133 [8/256/0/180] [13323.127553] LNet: Added LNI 192.168.206.117@tcp134 [8/256/0/180] [13323.931764] LNet: Added LNI 192.168.206.117@tcp135 [8/256/0/180] [13330.545074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13330.598047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13331.753811] Lustre: 463788:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.117@tcp: can't add NID 192.168.206.117@tcp130: rc = -75 [13331.811831] LNetError: 461245:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.117 [13331.820041] LNetError: Protocol error connecting to 192.168.206.117@tcp120 at host 192.168.206.117:988: is it running a compatible version of Lustre? [13331.836252] LNetError: Skipped 8 previous similar messages [13331.906201] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13331.938269] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13332.017647] Lustre: lustre-MDT0000: new disk, initializing [13332.090753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13332.107217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13332.835993] LNetError: 463817:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.117 [13332.851139] LNet: 5 local NIs in recovery (showing 5): 192.168.206.117@tcp126, 192.168.206.117@tcp127, 192.168.206.117@tcp128, 192.168.206.117@tcp129, 192.168.206.117@tcp120 [13332.858473] LNetError: 463817:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [13332.858603] LNetError: Protocol error connecting to 192.168.206.117@tcp101 at host 192.168.206.117:988: is it running a compatible version of Lustre? [13334.950329] LNetError: 463820:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.117 [13334.965692] LNetError: 463820:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [13334.979380] LNetError: Protocol error connecting to 192.168.206.117@tcp129 at host 192.168.206.117:988: is it running a compatible version of Lustre? [13334.993088] LNetError: Skipped 7 previous similar messages [13336.424717] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [13338.018832] LNetError: 463818:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.117 [13338.034787] LNetError: 463818:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [13338.043224] LNetError: Protocol error connecting to 192.168.206.117@tcp128 at host 192.168.206.117:988: is it running a compatible version of Lustre? [13346.212600] LNetError: 461245:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.117 [13346.231791] LNetError: Protocol error connecting to 192.168.206.117@tcp128 at host 192.168.206.117:988: is it running a compatible version of Lustre? [13347.294413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13347.440736] Lustre: 463806:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13347.499691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13347.504052] Lustre: Skipped 1 previous similar message [13347.574389] Lustre: lustre-MDT0001: new disk, initializing [13347.600798] Lustre: 464607:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.117@tcp: can't add NID 192.168.206.117@tcp130: rc = -75 [13347.607789] Lustre: 464607:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13347.711408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13347.752539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13347.765318] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13352.498204] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [13354.537307] Lustre: 464971:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.117@tcp: can't add NID 192.168.206.117@tcp130: rc = -75 [13354.551906] Lustre: 464971:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13361.571642] LNetError: 463817:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.117 [13361.586757] LNetError: Protocol error connecting to 192.168.206.117@tcp121 at host 192.168.206.117:988: is it running a compatible version of Lustre? [13362.324370] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13369.460982] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13378.980301] LNetError: 461246:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.117 [13378.991963] LNetError: 461246:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [13378.997841] LNetError: Protocol error connecting to 192.168.206.117@tcp127 at host 192.168.206.117:988: is it running a compatible version of Lustre? [13379.006615] LNetError: Skipped 11 previous similar messages [13379.662474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13380.821516] Lustre: lustre-OST0000: new disk, initializing [13380.835195] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13380.861436] Lustre: 465780:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13381.176603] Lustre: 465785:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.117@tcp: can't add NID 192.168.206.117@tcp130: rc = -75 [13381.204953] Lustre: 465785:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13381.325782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13388.608189] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [13389.356995] Lustre: 466159:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.117@tcp: can't add NID 192.168.206.117@tcp130: rc = -75 [13389.371489] Lustre: 466159:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13389.463690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13389.477556] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13389.614485] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13392.297100] LNet: 1 local NIs in recovery (showing 1): 192.168.206.117@tcp128 [13399.315240] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13410.278544] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13410.279748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13410.284072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13410.293447] LustreError: Skipped 1 previous similar message [13410.315242] Lustre: Skipped 1 previous similar message [13411.744903] LNetError: 463817:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.117 [13411.762344] LNetError: 463817:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 18 previous similar messages [13411.772120] LNetError: Protocol error connecting to 192.168.206.117@tcp126 at host 192.168.206.117:988: is it running a compatible version of Lustre? [13411.784720] LNetError: Skipped 18 previous similar messages [13412.811347] Lustre: server umount lustre-OST0000 complete [13419.489751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13419.499468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13419.526802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13419.535283] Lustre: Skipped 1 previous similar message [13420.516736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13420.553611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13420.561379] Lustre: Skipped 1 previous similar message [13425.634494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13425.641247] Lustre: Skipped 1 previous similar message [13430.763815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13430.782742] Lustre: Skipped 1 previous similar message [13431.263283] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13431.474252] Lustre: server umount lustre-MDT0000 complete [13435.245184] LustreError: 463798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842626 with bad export cookie 13347862412638002295 [13435.255634] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13435.261951] LustreError: 463798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13435.680504] Lustre: server umount lustre-MDT0001 complete [13447.297123] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [13450.310972] Key type lgssc unregistered [13450.617422] LNet: 467697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13450.629121] LNetError: 467697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13450.634958] LNet: Removed LNI 192.168.206.117@tcp [13451.679391] LNet: Removed LNI 192.168.206.117@tcp100 [13451.690186] LNetError: 467697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13451.702408] LNetError: 467697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13451.713278] LNet: Removed LNI 192.168.206.117@tcp101 [13452.767884] LNet: Removed LNI 192.168.206.117@tcp102 [13452.772820] LNetError: 467697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13452.777489] LNetError: 467697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13452.786795] LNet: Removed LNI 192.168.206.117@tcp103 [13453.855257] LNet: Removed LNI 192.168.206.117@tcp104 [13454.879619] LNet: Removed LNI 192.168.206.117@tcp105 [13454.888056] LNetError: 467697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13454.911439] LNetError: 467697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13454.921318] LNet: Removed LNI 192.168.206.117@tcp106 [13455.967151] LNet: Removed LNI 192.168.206.117@tcp107 [13456.991532] LNet: Removed LNI 192.168.206.117@tcp108 [13458.015258] LNet: Removed LNI 192.168.206.117@tcp109 [13459.039253] LNet: Removed LNI 192.168.206.117@tcp110 [13459.043205] LNetError: 467697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13459.051453] LNetError: 467697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13459.056422] LNet: Removed LNI 192.168.206.117@tcp111 [13460.131031] LNet: Removed LNI 192.168.206.117@tcp112 [13461.151128] LNet: Removed LNI 192.168.206.117@tcp113 [13462.175136] LNet: Removed LNI 192.168.206.117@tcp114 [13463.200756] LNet: Removed LNI 192.168.206.117@tcp115 [13464.223153] LNet: Removed LNI 192.168.206.117@tcp116 [13465.247368] LNet: Removed LNI 192.168.206.117@tcp117 [13466.271161] LNet: Removed LNI 192.168.206.117@tcp118 [13467.295243] LNet: Removed LNI 192.168.206.117@tcp119 [13467.310771] LNetError: 467697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13467.319688] LNetError: 467697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13467.325186] LNet: Removed LNI 192.168.206.117@tcp120 [13468.383145] LNet: Removed LNI 192.168.206.117@tcp121 [13469.407376] LNet: Removed LNI 192.168.206.117@tcp122 [13470.431165] LNet: Removed LNI 192.168.206.117@tcp123 [13471.455130] LNet: Removed LNI 192.168.206.117@tcp124 [13472.479429] LNet: Removed LNI 192.168.206.117@tcp125 [13473.503737] LNet: Removed LNI 192.168.206.117@tcp126 [13474.527218] LNet: Removed LNI 192.168.206.117@tcp127 [13474.537426] LNet: Removed LNI 192.168.206.117@tcp128 [13475.551156] LNet: Removed LNI 192.168.206.117@tcp129 [13475.556602] LNet: Removed LNI 192.168.206.117@tcp130 [13475.562650] LNet: Removed LNI 192.168.206.117@tcp131 [13475.570843] LNet: Removed LNI 192.168.206.117@tcp132 [13475.577485] LNet: Removed LNI 192.168.206.117@tcp133 [13475.581876] LNet: Removed LNI 192.168.206.117@tcp134 [13475.601095] LNet: Removed LNI 192.168.206.117@tcp135 [13476.782116] Key type .llcrypt unregistered [13476.784471] Key type ._llcrypt unregistered [13500.519454] Key type ._llcrypt registered [13500.523766] Key type .llcrypt registered [13500.648327] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [13516.666333] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [13518.287140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13518.330531] alg: No test for adler32 (adler32-zlib) [13519.478731] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [13519.773402] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [13521.480643] Key type lgssc registered [13522.590556] Lustre: Echo OBD driver; http://www.lustre.org/ [13560.317109] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 10:59:10 (1787842750) [13580.070932] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [13583.106897] Key type lgssc unregistered [13583.466597] LNet: 471789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13583.476727] LNetError: 471789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13583.488652] LNet: Removed LNI 192.168.206.117@tcp [13584.271374] Key type .llcrypt unregistered [13584.272784] Key type ._llcrypt unregistered [13601.647834] Key type ._llcrypt registered [13601.650666] Key type .llcrypt registered [13601.776519] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [13602.860504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13602.948750] alg: No test for adler32 (adler32-zlib) [13604.149808] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [13604.419550] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [13606.119298] Key type lgssc registered [13607.442073] Lustre: Echo OBD driver; http://www.lustre.org/ [13616.644593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13616.668493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13617.890355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13617.916660] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13617.991477] Lustre: lustre-MDT0000: new disk, initializing [13618.056459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13618.068590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13622.203636] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [13632.977844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13633.111789] Lustre: 473003:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13633.200713] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13633.207970] Lustre: Skipped 1 previous similar message [13633.278678] Lustre: lustre-MDT0001: new disk, initializing [13633.419721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13633.467657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13633.490388] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13639.050968] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [13649.709078] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13656.680496] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13663.642775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13664.013645] Lustre: lustre-OST0000: new disk, initializing [13664.019231] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13664.028424] Lustre: 474973:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13664.126831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13670.471633] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [13674.142432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13674.163388] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13674.311191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13681.359497] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13686.573074] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing lnet_if_list [13692.314440] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13692.371073] LNet: Added LNI 192.168.206.117@tcp10 [8/256/0/180] [13697.984161] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13698.009859] LNet: Added LNI 192.168.206.117@tcp11 [8/256/0/180] [13701.940112] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13701.995135] LNet: Added LNI 192.168.206.117@tcp12 [8/256/0/180] [13706.247943] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13706.272634] LNet: Added LNI 192.168.206.117@tcp13 [8/256/0/180] [13711.068917] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13711.118484] LNet: Added LNI 192.168.206.117@tcp14 [8/256/0/180] [13715.798389] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13715.829329] LNet: Added LNI 192.168.206.117@tcp15 [8/256/0/180] [13720.383657] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13720.421229] LNet: Added LNI 192.168.206.117@tcp16 [8/256/0/180] [13725.335254] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13725.362443] LNet: Added LNI 192.168.206.117@tcp17 [8/256/0/180] [13730.584955] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13730.629026] LNet: Added LNI 192.168.206.117@tcp18 [8/256/0/180] [13735.495858] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13735.530586] LNet: Added LNI 192.168.206.117@tcp19 [8/256/0/180] [13740.023843] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13740.070646] LNet: Added LNI 192.168.206.117@tcp20 [8/256/0/180] [13745.066907] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13745.093638] LNet: Added LNI 192.168.206.117@tcp21 [8/256/0/180] [13749.642417] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13749.667306] LNet: Added LNI 192.168.206.117@tcp22 [8/256/0/180] [13754.274285] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13754.316404] LNet: Added LNI 192.168.206.117@tcp23 [8/256/0/180] [13759.987496] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13760.068229] LNet: Added LNI 192.168.206.117@tcp24 [8/256/0/180] [13766.057400] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13766.121338] LNet: Added LNI 192.168.206.117@tcp25 [8/256/0/180] [13773.118275] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13773.146589] LNet: Added LNI 192.168.206.117@tcp26 [8/256/0/180] [13780.109099] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13780.159930] LNet: Added LNI 192.168.206.117@tcp27 [8/256/0/180] [13785.966996] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13785.990868] LNet: Added LNI 192.168.206.117@tcp28 [8/256/0/180] [13791.282937] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13791.305901] LNet: Added LNI 192.168.206.117@tcp29 [8/256/0/180] [13797.380633] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13797.435298] LNet: Added LNI 192.168.206.117@tcp30 [8/256/0/180] [13803.752258] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13803.775046] LNet: Added LNI 192.168.206.117@tcp31 [8/256/0/180] [13810.393445] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13810.437583] LNet: Added LNI 192.168.206.117@tcp32 [8/256/0/180] [13818.251433] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13818.296093] LNet: Added LNI 192.168.206.117@tcp33 [8/256/0/180] [13825.879766] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13825.911427] LNet: Added LNI 192.168.206.117@tcp34 [8/256/0/180] [13832.822388] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13832.902044] LNet: Added LNI 192.168.206.117@tcp35 [8/256/0/180] [13839.790918] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13839.837578] LNet: Added LNI 192.168.206.117@tcp36 [8/256/0/180] [13846.918219] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13846.950382] LNet: Added LNI 192.168.206.117@tcp37 [8/256/0/180] [13854.071556] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13854.100566] LNet: Added LNI 192.168.206.117@tcp38 [8/256/0/180] [13861.555828] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13861.591443] LNet: Added LNI 192.168.206.117@tcp39 [8/256/0/180] [13868.907874] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13868.936590] LNet: Added LNI 192.168.206.117@tcp40 [8/256/0/180] [13876.335884] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13876.394321] LNet: Added LNI 192.168.206.117@tcp41 [8/256/0/180] [13883.055895] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13883.106554] LNet: Added LNI 192.168.206.117@tcp42 [8/256/0/180] [13890.017672] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13890.054354] LNet: Added LNI 192.168.206.117@tcp43 [8/256/0/180] [13898.976186] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13899.015553] LNet: Added LNI 192.168.206.117@tcp44 [8/256/0/180] [13909.685732] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13909.720969] LNet: Added LNI 192.168.206.117@tcp45 [8/256/0/180] [13917.974855] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13918.021716] LNet: Added LNI 192.168.206.117@tcp46 [8/256/0/180] [13925.677784] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13925.702638] LNet: Added LNI 192.168.206.117@tcp47 [8/256/0/180] [13934.287528] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13934.319626] LNet: Added LNI 192.168.206.117@tcp48 [8/256/0/180] [13942.984681] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13943.022706] LNet: Added LNI 192.168.206.117@tcp49 [8/256/0/180] [13951.049916] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13951.083464] LNet: Added LNI 192.168.206.117@tcp50 [8/256/0/180] [13971.426225] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13971.426793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13971.433223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13971.440250] Lustre: Skipped 2 previous similar messages [13974.273563] Lustre: server umount lustre-MDT0000 complete [13977.615473] LustreError: 472996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843168 with bad export cookie 9454652608409734263 [13977.619322] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13977.624637] LustreError: 472996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13977.931758] Lustre: server umount lustre-MDT0001 complete [13995.367627] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [13997.023714] Lustre: 472276:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843172/real 1787843172] req@00000000d3ec06f2 x1874689046772992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787843188 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13997.068766] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13997.083517] Lustre: Skipped 2 previous similar messages [13998.113649] Lustre: 472276:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843173/real 1787843173] req@000000002bb2b3f1 x1874689046773120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787843189 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14002.145108] Lustre: 472275:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843177/real 1787843177] req@0000000020a89477 x1874689046773248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787843193 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14003.933417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14009.311182] Lustre: 472277:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843184/real 1787843184] req@000000005adcd586 x1874689046773632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787843200 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14009.337708] Lustre: 472277:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14023.058693] LustreError: 474966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14023.082226] LustreError: 474966:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14023.129205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14026.208848] LustreError: 474967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14026.225074] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14027.236159] LustreError: 474966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14027.652730] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [14032.354341] LustreError: 474967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14035.955156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14036.392153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14041.399277] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [14042.606952] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14048.727752] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14054.218463] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14072.296416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14072.309889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14072.322729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14073.324369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14073.335235] Lustre: Skipped 1 previous similar message [14077.245369] Lustre: server umount lustre-OST0000 complete [14082.532602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14082.539419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14082.551088] Lustre: Skipped 1 previous similar message [14082.554463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14086.548530] Lustre: server umount lustre-MDT0000 complete [14088.676293] LustreError: 499230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14088.690970] LustreError: 499230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14089.669597] LustreError: 472997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843280 with bad export cookie 9454652608409736881 [14089.672521] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14089.683132] LustreError: 472997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14089.902978] Lustre: server umount lustre-MDT0001 complete [14099.933519] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [14102.297209] Key type lgssc unregistered [14102.631534] LNet: 502009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14102.659577] LNetError: 502009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14102.671756] LNet: Removed LNI 192.168.206.117@tcp [14102.674797] LNet: Removed LNI 192.168.206.117@tcp10 [14102.679665] LNet: Removed LNI 192.168.206.117@tcp11 [14102.688896] LNet: Removed LNI 192.168.206.117@tcp12 [14102.693849] LNet: Removed LNI 192.168.206.117@tcp13 [14102.701380] LNet: Removed LNI 192.168.206.117@tcp14 [14102.703141] LNet: Removed LNI 192.168.206.117@tcp15 [14102.713174] LNet: Removed LNI 192.168.206.117@tcp16 [14102.718886] LNet: Removed LNI 192.168.206.117@tcp17 [14102.722655] LNet: Removed LNI 192.168.206.117@tcp18 [14102.734229] LNet: Removed LNI 192.168.206.117@tcp19 [14102.741740] LNet: Removed LNI 192.168.206.117@tcp20 [14102.748198] LNet: Removed LNI 192.168.206.117@tcp21 [14102.752038] LNet: Removed LNI 192.168.206.117@tcp22 [14102.754848] LNet: Removed LNI 192.168.206.117@tcp23 [14102.757946] LNet: Removed LNI 192.168.206.117@tcp24 [14102.761593] LNet: Removed LNI 192.168.206.117@tcp25 [14102.763657] LNet: Removed LNI 192.168.206.117@tcp26 [14102.765715] LNet: Removed LNI 192.168.206.117@tcp27 [14102.767921] LNet: Removed LNI 192.168.206.117@tcp28 [14102.769922] LNet: Removed LNI 192.168.206.117@tcp29 [14102.771767] LNet: Removed LNI 192.168.206.117@tcp30 [14102.773256] LNet: Removed LNI 192.168.206.117@tcp31 [14102.774695] LNet: Removed LNI 192.168.206.117@tcp32 [14102.776195] LNet: Removed LNI 192.168.206.117@tcp33 [14102.777676] LNet: Removed LNI 192.168.206.117@tcp34 [14102.779639] LNet: Removed LNI 192.168.206.117@tcp35 [14102.781854] LNet: Removed LNI 192.168.206.117@tcp36 [14102.783956] LNet: Removed LNI 192.168.206.117@tcp37 [14102.785935] LNet: Removed LNI 192.168.206.117@tcp38 [14102.790182] LNet: Removed LNI 192.168.206.117@tcp39 [14102.791912] LNet: Removed LNI 192.168.206.117@tcp40 [14102.794412] LNet: Removed LNI 192.168.206.117@tcp41 [14102.797030] LNet: Removed LNI 192.168.206.117@tcp42 [14102.799562] LNet: Removed LNI 192.168.206.117@tcp43 [14102.802245] LNet: Removed LNI 192.168.206.117@tcp44 [14102.804198] LNet: Removed LNI 192.168.206.117@tcp45 [14102.807086] LNet: Removed LNI 192.168.206.117@tcp46 [14102.809591] LNet: Removed LNI 192.168.206.117@tcp47 [14102.812131] LNet: Removed LNI 192.168.206.117@tcp48 [14102.814490] LNet: Removed LNI 192.168.206.117@tcp49 [14102.826288] LNet: Removed LNI 192.168.206.117@tcp50 [14103.424786] Key type .llcrypt unregistered [14103.428862] Key type ._llcrypt unregistered [14123.144736] Key type ._llcrypt registered [14123.149369] Key type .llcrypt registered [14123.230435] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [14136.259823] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [14137.220867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14137.313097] alg: No test for adler32 (adler32-zlib) [14138.369894] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [14138.583201] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [14140.248308] Key type lgssc registered [14141.196221] Lustre: Echo OBD driver; http://www.lustre.org/ [14175.346411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14175.392314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14176.711360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14176.771716] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14176.890991] Lustre: lustre-MDT0000: new disk, initializing [14176.970677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14177.000468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14181.320617] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [14192.758101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14192.936792] Lustre: 505183:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14193.035291] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14193.041738] Lustre: Skipped 1 previous similar message [14193.119984] Lustre: lustre-MDT0001: new disk, initializing [14193.238618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14193.303535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14193.320571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14198.211186] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [14207.625809] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14213.408359] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14220.181313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14220.419578] Lustre: *** cfs_fail_loc=90f, val=0*** [14220.483549] Lustre: lustre-OST0000: new disk, initializing [14220.489650] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14220.495541] Lustre: 507200:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14220.597965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14225.836224] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [14226.496224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14226.505677] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14226.575491] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14234.837640] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14242.276749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14242.277306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14242.281250] LustreError: Skipped 1 previous similar message [14242.284454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14242.291502] Lustre: Skipped 1 previous similar message [14242.314986] Lustre: Skipped 1 previous similar message [14247.392869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14247.400915] Lustre: Skipped 1 previous similar message [14247.729970] Lustre: server umount lustre-OST0000 complete [14252.518328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14252.535734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14252.559551] Lustre: Skipped 1 previous similar message [14257.451794] Lustre: server umount lustre-MDT0000 complete [14260.511495] LustreError: 505176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843451 with bad export cookie 2189565254411362635 [14260.528636] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14260.532252] LustreError: 505176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14260.803418] Lustre: server umount lustre-MDT0001 complete [14268.980044] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [14271.157170] Key type lgssc unregistered [14271.393680] LNet: 509107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14271.403632] LNetError: 509107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14271.420589] LNet: Removed LNI 192.168.206.117@tcp [14272.082958] Key type .llcrypt unregistered [14272.086385] Key type ._llcrypt unregistered [14289.436583] Key type ._llcrypt registered [14289.439206] Key type .llcrypt registered [14289.551878] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [14290.610682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14290.632626] alg: No test for adler32 (adler32-zlib) [14291.728566] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [14292.027210] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [14293.703151] Key type lgssc registered [14294.582136] Lustre: Echo OBD driver; http://www.lustre.org/ [14302.822724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14302.838626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14304.300632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14308.721479] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [14314.473035] LustreError: 510380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14314.493482] LustreError: 510380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14316.857504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14317.185203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14321.369502] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [14329.118213] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14334.772549] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14340.931282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14341.297915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14347.534715] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [14354.292519] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14365.054604] loop0: detected capacity change from 0 to 204800000 [14365.252143] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14365.274907] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14365.287807] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14371.200861] loop0: detected capacity change from 0 to 204800000 [14371.366976] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14371.375727] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14371.382535] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14376.022327] loop0: detected capacity change from 0 to 204800000 [14380.940509] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14381.067135] Lustre: fs15246-OST0000: new disk, initializing [14381.069935] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14381.074630] Lustre: 513973:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14381.126836] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14386.245487] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [14395.422339] loop1: detected capacity change from 0 to 204800000 [14399.817566] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14399.909947] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14399.961472] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14400.030466] Lustre: fs15246-MDT0000: new disk, initializing [14400.168761] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14400.186926] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14400.270480] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14401.314627] Lustre: 515094:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000b20286f4 x1874689768075520/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787843608 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14403.673451] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [14410.409468] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14410.416441] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14410.440364] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14424.554588] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14424.581752] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14425.830367] Lustre: server umount fs15246-MDT0000 complete [14429.675826] LustreError: 512218:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14432.919710] Lustre: server umount fs15246-OST0000 complete [14445.034051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14445.035627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14445.041030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14445.059120] Lustre: Skipped 1 previous similar message [14450.146129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14450.160205] Lustre: Skipped 2 previous similar messages [14455.268529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14455.277115] Lustre: Skipped 1 previous similar message [14456.287397] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14456.435173] Lustre: server umount lustre-OST0000 complete [14460.388836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14460.390119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14460.402687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14465.940764] Lustre: server umount lustre-MDT0000 complete [14469.428415] LustreError: 510361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843660 with bad export cookie 608876455229507580 [14469.431970] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14469.442482] LustreError: 510361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14469.776565] Lustre: server umount lustre-MDT0001 complete [14482.140514] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [14484.707658] Key type lgssc unregistered [14484.959917] LNet: 518195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14484.975995] LNetError: 518195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14484.999783] LNet: Removed LNI 192.168.206.117@tcp [14485.726449] Key type .llcrypt unregistered [14485.728331] Key type ._llcrypt unregistered [14505.205894] Key type ._llcrypt registered [14505.208040] Key type .llcrypt registered [14505.357380] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [14506.949027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14506.983652] alg: No test for adler32 (adler32-zlib) [14508.107682] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [14508.342273] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [14510.039180] Key type lgssc registered [14511.697021] Lustre: Echo OBD driver; http://www.lustre.org/ [14551.463446] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [14561.019650] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [14600.633438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14600.670586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14601.959506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14602.010086] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14602.084535] Lustre: lustre-MDT0000: new disk, initializing [14602.147778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14602.168106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14606.622316] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [14616.603668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14616.677603] Lustre: 522662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14616.754656] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14616.760149] Lustre: Skipped 1 previous similar message [14616.869269] Lustre: lustre-MDT0001: new disk, initializing [14617.015125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14617.067265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14617.079768] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14620.943534] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [14630.184175] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14636.547873] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14643.772561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14643.987138] Lustre: lustre-OST0000: new disk, initializing [14643.996758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14644.007224] Lustre: 524632:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14644.095318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14650.036761] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [14653.027173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14653.043644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14653.148017] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14659.843288] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14665.445420] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14665.712491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14670.063845] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14670.268741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14673.894489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14673.898772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14673.902015] LustreError: Skipped 1 previous similar message [14673.904353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14673.942983] Lustre: Skipped 1 previous similar message [14678.474543] Lustre: server umount lustre-OST0000 complete [14683.622683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14683.633567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14683.653162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14683.664766] Lustre: Skipped 1 previous similar message [14688.500949] Lustre: server umount lustre-MDT0000 complete [14692.018824] LustreError: 523459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843883 with bad export cookie 8051081330243083087 [14692.022339] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14692.028660] LustreError: 523459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14692.487394] Lustre: server umount lustre-MDT0001 complete [14718.655428] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [14727.401794] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [14760.820671] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14480 sec ========= 11:19:10 (1787843950) [14762.613242] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:19:12 (1787843952) === [14766.140872] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:19:16 (1787843956) === [14806.727319] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [14809.651996] Key type lgssc unregistered [14809.937574] LNet: 531281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14809.948212] LNetError: 531281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14809.961907] LNet: Removed LNI 192.168.206.117@tcp [14810.602599] Key type .llcrypt unregistered [14810.606827] Key type ._llcrypt unregistered