[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 544420418 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001024] APIC: Switch to symmetric I/O mode setup [ 0.003268] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008537] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011141] LSM: Security Framework initializing [ 0.012047] Yama: becoming mindful. [ 0.013036] SELinux: Initializing. [ 0.014066] *** VALIDATE selinux *** [ 0.021724] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026359] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027164] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028102] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029156] *** VALIDATE tmpfs *** [ 0.031177] *** VALIDATE proc *** [ 0.032214] *** VALIDATE cgroup *** [ 0.033005] *** VALIDATE cgroup2 *** [ 0.034274] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037034] Spectre V2 : User space: Vulnerable [ 0.038006] Speculative Store Bypass: Vulnerable [ 0.041092] debug: unmapping init [mem 0xffffffff86059000-0xffffffff86060fff] [ 0.043969] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044801] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045039] ... version: 2 [ 0.046012] ... bit width: 48 [ 0.047011] ... generic registers: 4 [ 0.048008] ... value mask: 0000ffffffffffff [ 0.049009] ... max period: 00007fffffffffff [ 0.050014] ... fixed-purpose events: 3 [ 0.051008] ... event mask: 000000070000000f [ 0.052313] rcu: Hierarchical SRCU implementation. [ 0.054544] smp: Bringing up secondary CPUs ... [ 0.055550] x86: Booting SMP configuration: [ 0.056025] .... node #0, CPUs: #1 #2 #3 [ 0.068022] smp: Brought up 1 node, 4 CPUs [ 0.070128] smpboot: Max logical packages: 1 [ 0.071132] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.156970] node 0 deferred pages initialised in 83ms [ 0.165039] devtmpfs: initialized [ 0.166246] x86/mm: Memory block size: 128MB [ 0.170055] gcov: version magic: 0x41383552 [ 0.172501] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.173182] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.174305] pinctrl core: initialized pinctrl subsystem [ 0.175300] [ 0.175880] ************************************************************* [ 0.176012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.177011] ** ** [ 0.178009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.179015] ** ** [ 0.180025] ** This means that this kernel is built to expose internal ** [ 0.181009] ** IOMMU data structures, which may compromise security on ** [ 0.182011] ** your system. ** [ 0.183011] ** ** [ 0.184009] ** If you see this message and you are not debugging the ** [ 0.185008] ** kernel, report this immediately to your vendor! ** [ 0.186008] ** ** [ 0.187007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.188008] ************************************************************* [ 0.190202] NET: Registered protocol family 16 [ 0.191488] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.192067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.193093] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.195086] cpuidle: using governor menu [ 0.197727] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.202631] PCI: Using configuration type 1 for base access [ 0.203126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.212108] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.213000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.214346] cryptd: max_cpu_qlen set to 1000 [ 0.216000] ACPI: Added _OSI(Module Device) [ 0.218018] ACPI: Added _OSI(Processor Device) [ 0.220032] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.223014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.231126] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.241227] ACPI: Interpreter enabled [ 0.242096] ACPI: PM: (supports S0 S3 S4 S5) [ 0.243013] ACPI: Using IOAPIC for interrupt routing [ 0.244126] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.250607] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.264356] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.267046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.270029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.277028] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.284085] acpiphp: Slot [2] registered [ 0.286119] acpiphp: Slot [5] registered [ 0.288151] acpiphp: Slot [6] registered [ 0.290204] acpiphp: Slot [7] registered [ 0.292194] acpiphp: Slot [8] registered [ 0.296279] acpiphp: Slot [9] registered [ 0.297319] acpiphp: Slot [10] registered [ 0.300164] acpiphp: Slot [3] registered [ 0.305152] acpiphp: Slot [4] registered [ 0.307320] acpiphp: Slot [11] registered [ 0.315222] acpiphp: Slot [12] registered [ 0.316243] acpiphp: Slot [13] registered [ 0.319375] acpiphp: Slot [14] registered [ 0.324191] acpiphp: Slot [15] registered [ 0.327149] acpiphp: Slot [16] registered [ 0.329123] acpiphp: Slot [17] registered [ 0.330082] acpiphp: Slot [18] registered [ 0.331072] acpiphp: Slot [19] registered [ 0.333068] acpiphp: Slot [20] registered [ 0.334075] acpiphp: Slot [21] registered [ 0.335082] acpiphp: Slot [22] registered [ 0.337075] acpiphp: Slot [23] registered [ 0.338070] acpiphp: Slot [24] registered [ 0.340113] acpiphp: Slot [25] registered [ 0.341082] acpiphp: Slot [26] registered [ 0.343107] acpiphp: Slot [27] registered [ 0.345209] acpiphp: Slot [28] registered [ 0.346109] acpiphp: Slot [29] registered [ 0.348085] acpiphp: Slot [30] registered [ 0.349092] acpiphp: Slot [31] registered [ 0.351072] PCI host bridge to bus 0000:00 [ 0.353019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.355035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.359046] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.361027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.364019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.367090] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.369176] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.373103] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.378483] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.385018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.391464] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.394020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.397029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.400019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.403594] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.405920] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.409048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.412922] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.418015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.432017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.438017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.454040] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.461019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.488019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.518015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.532560] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.547016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.554016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.585015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.602498] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.621016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.631018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.699016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.715327] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.732022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.744016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.769015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.778000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.785017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.801025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.863018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.875996] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.887016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.896017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.907019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.922000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.925382] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.928326] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.931327] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.933166] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.937215] iommu: Default domain type: Passthrough [ 0.938436] SCSI subsystem initialized [ 0.939000] ACPI: bus type USB registered [ 0.939145] usbcore: registered new interface driver usbfs [ 0.940076] usbcore: registered new interface driver hub [ 0.942038] usbcore: registered new device driver usb [ 0.943172] pps_core: LinuxPPS API ver. 1 registered [ 0.946010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.949063] PTP clock support registered [ 0.952473] EDAC MC: Ver: 3.0.0 [ 0.956558] PCI: Using ACPI for IRQ routing [ 0.966146] NetLabel: Initializing [ 0.970021] NetLabel: domain hash size = 128 [ 0.972020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.978338] NetLabel: unlabeled traffic allowed by default [ 0.984671] vgaarb: loaded [ 0.988109] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.992032] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.005000] clocksource: Switched to clocksource kvm-clock [ 1.194614] VFS: Disk quotas dquot_6.6.0 [ 1.197202] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.203046] *** VALIDATE ramfs *** [ 1.205477] *** VALIDATE hugetlbfs *** [ 1.208823] pnp: PnP ACPI init [ 1.212275] pnp: PnP ACPI: found 6 devices [ 1.228609] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.232753] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.235064] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.237700] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.239893] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.241926] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.244377] NET: Registered protocol family 2 [ 1.246860] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.251742] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.255075] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.260337] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.264374] TCP: Hash tables configured (established 65536 bind 65536) [ 1.267089] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.270057] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.272983] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.276349] NET: Registered protocol family 1 [ 1.278919] RPC: Registered named UNIX socket transport module. [ 1.281071] RPC: Registered udp transport module. [ 1.283108] RPC: Registered tcp transport module. [ 1.284896] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.287115] NET: Registered protocol family 44 [ 1.288592] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.290398] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.292785] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.295053] PCI: CLS 0 bytes, default 64 [ 1.296820] Unpacking initramfs... [ 3.050278] debug: unmapping init [mem 0xffff9875bcc54000-0xffff9875bffbffff] [ 3.074697] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.076873] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.079517] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.912799] Initialise system trusted keyrings [ 3.914293] Key type blacklist registered [ 3.916415] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.926256] zbud: loaded [ 3.929377] *** VALIDATE nfs *** [ 3.930553] *** VALIDATE nfs4 *** [ 3.932273] pstore: using deflate compression [ 3.936121] Platform Keyring initialized [ 4.066270] NET: Registered protocol family 38 [ 4.068962] Key type asymmetric registered [ 4.070507] Asymmetric key parser 'x509' registered [ 4.073299] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.077126] io scheduler mq-deadline registered [ 4.079647] io scheduler kyber registered [ 4.081944] io scheduler bfq registered [ 4.083820] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.088526] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.091794] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.095302] ACPI: Power Button [PWRF] [ 4.104676] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.111841] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.125361] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.132146] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.146469] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.175483] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.204682] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.208291] Non-volatile memory driver v1.3 [ 4.209408] Linux agpgart interface v0.103 [ 4.243796] virtio_blk virtio1: [vda] 145928 512-byte logical blocks (74.7 MB/71.3 MiB) [ 4.246500] vda: detected capacity change from 0 to 74715136 [ 4.267221] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.270712] vdb: detected capacity change from 0 to 1073741824 [ 4.288320] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.291794] vdc: detected capacity change from 0 to 2621440000 [ 4.314990] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.318212] vdd: detected capacity change from 0 to 2621440000 [ 4.341194] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.344354] vde: detected capacity change from 0 to 4294967296 [ 4.366751] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.369348] vdf: detected capacity change from 0 to 4294967296 [ 4.379472] libphy: Fixed MDIO Bus: probed [ 4.393237] usbcore: registered new interface driver usbserial_generic [ 4.395782] usbserial: USB Serial support registered for generic [ 4.398066] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.402267] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.404277] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.406692] mousedev: PS/2 mouse device common for all mice [ 4.410137] rtc_cmos 00:05: RTC can wake from S4 [ 4.413898] rtc_cmos 00:05: registered as rtc0 [ 4.415614] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.418130] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.610488] intel_pstate: CPU model not supported [ 4.612474] hid: raw HID events driver (C) Jiri Kosina [ 4.619237] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.621845] usbcore: registered new interface driver usbhid [ 4.626865] usbhid: USB HID core driver [ 4.628200] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.629342] drop_monitor: Initializing network drop monitor service [ 4.629494] Initializing XFRM netlink socket [ 4.631142] NET: Registered protocol family 10 [ 4.643018] Segment Routing with IPv6 [ 4.644449] NET: Registered protocol family 17 [ 4.646416] mpls_gso: MPLS GSO support [ 4.652874] RAS: Correctable Errors collector initialized. [ 4.655049] AVX version of gcm_enc/dec engaged. [ 4.656730] AES CTR mode by8 optimization enabled [ 4.736665] sched_clock: Marking stable (4736649157, 0)->(5947203580, -1210554423) [ 4.740612] registered taskstats version 1 [ 4.742356] Loading compiled-in X.509 certificates [ 4.743960] zswap: loaded using pool lzo/zbud [ 4.778474] Key type big_key registered [ 4.793582] Key type encrypted registered [ 4.794794] ima: No TPM chip found, activating TPM-bypass! [ 4.797348] ima: Allocated hash algorithm: sha1 [ 4.799529] ima: No architecture policies found [ 4.801260] evm: Initialising EVM extended attributes: [ 4.802757] evm: security.selinux [ 4.804389] evm: security.ima [ 4.805259] evm: security.capability [ 4.806916] evm: HMAC attrs: 0x1 [ 4.809513] rtc_cmos 00:05: setting system clock to 2026-08-12 13:34:01 UTC (1786541641) [ 4.818516] debug: unmapping init [mem 0xffffffff87003000-0xffffffff871fffff] [ 4.821899] debug: unmapping init [mem 0xffffffff85d82000-0xffffffff86058fff] [ 4.831067] Write protecting the kernel read-only data: 28672k [ 4.836769] debug: unmapping init [mem 0xffffffff84403000-0xffffffff845fffff] [ 4.843157] debug: unmapping init [mem 0xffffffff84d14000-0xffffffff84dfffff] [ 4.948819] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.955715] systemd[1]: Detected virtualization kvm. [ 4.957486] systemd[1]: Detected architecture x86-64. [ 4.959493] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.996926] systemd[1]: No hostname configured. [ 4.999275] systemd[1]: Set hostname to . [ 5.001808] random: systemd: uninitialized urandom read (16 bytes read) [ 5.004410] systemd[1]: Initializing machine ID from random generator. [ 5.131613] random: ln: uninitialized urandom read (6 bytes read) [ 5.287975] random: systemd: uninitialized urandom read (16 bytes read) [ 5.290017] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 5.294640] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 5.298460] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.131991] device-mapper: uevent: version 1.0.3 [ 6.137007] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] [ 7.325837] random: fast init done Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.401482] virtio_net virtio0 ens2: renamed from eth0 [ 7.542792] scsi host0: ata_piix [ 7.578613] scsi host1: ata_piix [ 7.581294] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.585918] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.315229] random: crng init done [ 12.322529] random: 7 urandom warning(s) missed due to ratelimiting [ 13.276273] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 14.505042] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.206933] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.805332] SELinux: Disabled at runtime. [ 16.901743] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.912368] systemd[1]: Detected virtualization kvm. [ 16.914690] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.995422] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.002563] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.011715] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.017790] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.024696] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.046844] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.059913] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current k[ 18.280501] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 19.000874] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.531391] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.584981] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.817897] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.858635] EDAC sbridge: Ver: 1.1.2 [ 22.674686] Key type dns_resolver registered [ 23.135633] NFS: Registering the id_resolver key type [ 23.144612] Key type id_resolver registered [ 23.147395] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg359-server login: [ 49.981018] libcfs: loading out-of-tree module taints kernel. [ 49.999160] Key type ._llcrypt registered [ 50.000477] Key type .llcrypt registered [ 50.038285] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 57.312884] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 57.901164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 57.907199] alg: No test for adler32 (adler32-zlib) [ 58.905175] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 59.191826] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 60.799176] Key type lgssc registered [ 61.299762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 67.597517] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 81.867459] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 86.521698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 86.532066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 87.626026] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 87.637438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 87.672444] Lustre: lustre-MDT0000: new disk, initializing [ 87.693198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 87.700702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 89.110151] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 94.284448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 94.323753] Lustre: 6481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 94.335608] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 94.338527] Lustre: Skipped 1 previous similar message [ 94.377670] Lustre: lustre-MDT0001: new disk, initializing [ 94.399348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 94.407906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 94.412539] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 95.800884] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 98.038683] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 101.275609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 101.362847] Lustre: lustre-OST0000: new disk, initializing [ 101.364694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 101.367135] Lustre: 8414:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 101.385973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 101.750646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 101.753416] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 101.782166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 103.248388] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 108.563381] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 108.613941] Lustre: lustre-OST0001: new disk, initializing [ 108.616464] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 108.619314] Lustre: 9491:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 108.641151] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 108.837692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 108.841420] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 108.861539] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 110.774624] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 116.535264] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 120.145527] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 122.016193] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing check_logdir /tmp/testlogs/ [ 123.690429] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing yml_node [ 125.256353] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 126.145631] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 127.043904] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 127.615656] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 12 09:36:04 EDT 2026 [ 133.279337] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 133.818595] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 145.887499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 145.891393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 145.897209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 149.983881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 149.983945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 149.987971] Lustre: Skipped 1 previous similar message [ 149.989721] Lustre: Skipped 1 previous similar message [ 151.710734] Lustre: server umount lustre-MDT0000 complete [ 152.952694] LustreError: 6474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786541789 with bad export cookie 7021145264821780431 [ 152.954119] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 152.958103] LustreError: 6474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 155.103697] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 155.103902] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 155.109492] Lustre: Skipped 2 previous similar messages [ 155.112724] Lustre: Skipped 2 previous similar messages [ 157.151689] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 157.154741] Lustre: Skipped 1 previous similar message [ 162.271501] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 162.271514] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 162.277254] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 162.280109] Lustre: Skipped 1 previous similar message [ 163.295656] LustreError: 6491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 163.302779] LustreError: 6491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 167.391107] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 167.460788] Lustre: server umount lustre-MDT0001 complete [ 183.263114] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 183.295602] Lustre: server umount lustre-OST0000 complete [ 190.735291] Lustre: server umount lustre-OST0001 complete [ 192.944172] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 196.301129] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 209.278040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.367092] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 209.380062] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 209.418479] Lustre: lustre-MDT0000: new disk, initializing [ 209.444871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.453775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 210.892362] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 215.403454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 215.452210] Lustre: 15034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 215.564655] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 217.034707] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 221.048436] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 223.127268] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 225.527231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 225.637589] Lustre: lustre-OST0000: new disk, initializing [ 225.640156] Lustre: Skipped 1 previous similar message [ 225.642658] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 225.645728] Lustre: Skipped 2 previous similar messages [ 225.649134] Lustre: 16999:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 225.682392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 225.685428] Lustre: Skipped 1 previous similar message [ 226.885364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 226.888350] Lustre: Skipped 1 previous similar message [ 226.890761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 226.931277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 227.854171] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 231.716653] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 233.235830] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 233.321098] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 234.757794] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 234.834901] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 237.024187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 237.028380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 237.034330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 237.037435] Lustre: Skipped 2 previous similar messages [ 241.632452] Lustre: server umount lustre-OST0000 complete [ 246.240304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 246.244318] LustreError: Skipped 1 previous similar message [ 246.246418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 246.251777] Lustre: Skipped 1 previous similar message [ 249.182052] Lustre: server umount lustre-MDT0000 complete [ 250.626071] LustreError: 15827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786541887 with bad export cookie 7021145264821781894 [ 250.630545] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 250.633038] LustreError: 15827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 250.757535] Lustre: server umount lustre-MDT0001 complete [ 252.301840] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:38:08 (1786541888) [ 253.250606] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 253.897530] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:38:10 (1786541890) [ 256.507063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 256.687390] LustreError: 19405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 256.718349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 258.255362] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 261.544637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 263.299193] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 266.094246] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 268.229104] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 270.826960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 273.278865] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 276.028532] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 292.831924] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 292.835277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 292.840873] Lustre: Skipped 1 previous similar message [ 292.843372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 292.846054] Lustre: Skipped 4 previous similar messages [ 295.510561] Lustre: server umount lustre-OST0000 complete [ 297.441724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 297.445564] LustreError: Skipped 1 previous similar message [ 311.263115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 311.323948] Lustre: server umount lustre-MDT0000 complete [ 312.595582] LustreError: 19385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786541949 with bad export cookie 7021145264821782776 [ 312.597810] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 312.601684] LustreError: 19385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 312.726635] Lustre: server umount lustre-MDT0001 complete [ 316.638170] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 317.837217] Key type lgssc unregistered [ 317.972363] LNet: 22959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.975624] LNetError: 22959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 317.988416] LNet: Removed LNI 192.168.203.159@tcp [ 318.342106] Key type .llcrypt unregistered [ 318.343414] Key type ._llcrypt unregistered [ 327.538133] Key type ._llcrypt registered [ 327.539787] Key type .llcrypt registered [ 327.579210] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 327.982943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 327.994361] alg: No test for adler32 (adler32-zlib) [ 328.857087] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 328.946172] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 330.527153] Key type lgssc registered [ 330.923971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 333.812586] loop: module loaded [ 333.842475] loop0: detected capacity change from 0 to 204800000 [ 333.905303] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 333.912256] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 334.032372] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 336.182923] loop0: detected capacity change from 0 to 204800000 [ 336.245910] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 336.250423] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 336.255286] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 338.057621] loop0: detected capacity change from 0 to 204800000 [ 340.041278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 340.047610] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 341.130986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 341.146314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 341.184840] Lustre: lustre-MDT0000: new disk, initializing [ 341.208709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 341.216787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 342.606973] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 346.287571] loop1: detected capacity change from 0 to 204800000 [ 348.186427] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 348.306029] Lustre: lustre-OST07c6: new disk, initializing [ 348.308211] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 348.310519] Lustre: Skipped 1 previous similar message [ 348.313655] Lustre: 26125:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 348.342973] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 350.268327] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 356.863731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 356.866702] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 356.875513] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 362.919703] Lustre: 25111:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 362.923356] Lustre: 25111:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 362.926890] Lustre: 25111:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 362.929926] Lustre: 25111:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 362.932769] Lustre: 25111:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 362.935628] Lustre: 25111:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 376.311728] Lustre: 26133:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 376.315893] Lustre: 26133:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 376.318601] Lustre: 26133:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 376.321761] Lustre: 26133:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 376.324383] Lustre: 26133:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 376.326610] Lustre: 26133:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 379.812432] Lustre: 25113:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 379.815702] Lustre: 25113:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 379.818276] Lustre: 25113:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 379.820950] Lustre: 25113:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 379.824128] Lustre: 25113:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 379.827069] Lustre: 25113:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 379.829911] Lustre: 25113:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 379.832776] Lustre: 25113:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 379.835982] Lustre: 25113:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 379.838600] Lustre: 25113:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 379.841477] Lustre: 25113:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 379.844574] Lustre: 25113:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 382.902983] Lustre: 26133:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 382.905898] Lustre: 26133:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 382.908723] Lustre: 26133:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 382.911209] Lustre: 26133:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 382.913661] Lustre: 26133:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 382.916204] Lustre: 26133:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 382.918426] Lustre: 26133:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 382.921216] Lustre: 26133:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 382.923575] Lustre: 26133:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 382.926144] Lustre: 26133:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 382.929522] Lustre: 26133:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 382.933070] Lustre: 26133:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 389.461499] Lustre: 26133:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 389.464720] Lustre: 26133:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 389.467476] Lustre: 26133:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 389.470421] Lustre: 26133:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 389.473360] Lustre: 26133:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 389.476606] Lustre: 26133:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 389.479617] Lustre: 26133:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 389.482136] Lustre: 26133:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 389.484390] Lustre: 26133:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 389.486575] Lustre: 26133:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 389.489216] Lustre: 26133:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 389.491286] Lustre: 26133:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 399.020231] Lustre: 26133:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 399.023103] Lustre: 26133:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 399.025305] Lustre: 26133:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 399.027858] Lustre: 26133:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 399.030351] Lustre: 26133:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 399.033209] Lustre: 26133:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 399.035463] Lustre: 26133:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 399.038081] Lustre: 26133:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 399.040914] Lustre: 26133:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 399.043232] Lustre: 26133:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 399.045431] Lustre: 26133:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 399.048031] Lustre: 26133:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 439.790863] Lustre: server umount lustre-OST07c6 complete [ 453.138566] Lustre: server umount lustre-MDT0000 complete [ 458.646659] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 459.632640] Key type lgssc unregistered [ 459.760416] LNet: 28724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 459.763448] LNetError: 28724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 459.770294] LNet: Removed LNI 192.168.203.159@tcp [ 460.076089] Key type .llcrypt unregistered [ 460.077917] Key type ._llcrypt unregistered [ 467.336105] Key type ._llcrypt registered [ 467.337679] Key type .llcrypt registered [ 467.374415] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 467.684482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 467.721043] alg: No test for adler32 (adler32-zlib) [ 468.574773] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 468.671960] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 470.255146] Key type lgssc registered [ 470.572227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 472.957704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 472.963512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 474.089314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 475.309141] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 478.086307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 478.206752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 479.534824] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 481.840451] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 483.647580] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 485.833126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 485.950604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 487.774339] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 490.199664] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 500.328244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 505.825831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 505.827216] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 505.827375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 505.830174] Lustre: Skipped 1 previous similar message [ 510.944689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 510.947543] Lustre: Skipped 2 previous similar messages [ 511.119429] Lustre: server umount lustre-OST0000 complete [ 514.016106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 514.018699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.022755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 516.064194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 516.064837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 518.552775] Lustre: server umount lustre-MDT0000 complete [ 519.745788] LustreError: 29955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786542156 with bad export cookie 8035096012389363844 [ 519.746770] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 519.749780] LustreError: 29955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 519.863511] Lustre: server umount lustre-MDT0001 complete [ 523.293277] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 524.316875] Key type lgssc unregistered [ 524.439345] LNet: 33727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 524.442160] LNetError: 33727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 524.451313] LNet: Removed LNI 192.168.203.159@tcp [ 524.752056] Key type .llcrypt unregistered [ 524.753353] Key type ._llcrypt unregistered [ 531.839350] Key type ._llcrypt registered [ 531.840436] Key type .llcrypt registered [ 531.874664] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 532.247116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 532.262446] alg: No test for adler32 (adler32-zlib) [ 533.102217] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 533.188037] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 534.767203] Key type lgssc registered [ 535.114407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 537.490601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 537.497293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 538.624248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 539.843671] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 542.512398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.628551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 543.937561] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 546.241639] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 547.974922] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 550.115446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 550.222190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 552.142586] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 554.521486] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 563.558095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 568.799580] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 568.801016] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 568.803171] LustreError: Skipped 1 previous similar message [ 568.808505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 568.811549] Lustre: Skipped 1 previous similar message [ 573.922180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 573.924103] Lustre: Skipped 1 previous similar message [ 579.040740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 579.042298] Lustre: Skipped 1 previous similar message [ 583.135144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 583.184185] Lustre: server umount lustre-OST0000 complete [ 588.767816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 588.770253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.775980] Lustre: Skipped 1 previous similar message [ 588.777683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 590.619397] Lustre: server umount lustre-MDT0000 complete [ 591.905468] LustreError: 35706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786542228 with bad export cookie 10614662933013771674 [ 591.908574] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 591.909862] LustreError: 35706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 592.020386] Lustre: server umount lustre-MDT0001 complete [ 595.419783] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 596.447498] Key type lgssc unregistered [ 596.570426] LNet: 38527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 596.573256] LNetError: 38527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 596.581288] LNet: Removed LNI 192.168.203.159@tcp [ 596.859318] Key type .llcrypt unregistered [ 596.861047] Key type ._llcrypt unregistered [ 605.170504] Key type ._llcrypt registered [ 605.172183] Key type .llcrypt registered [ 605.209138] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 605.495349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 605.527266] alg: No test for adler32 (adler32-zlib) [ 606.367825] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 606.447612] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 608.023137] Key type lgssc registered [ 608.352658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 610.887941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 610.896285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 612.045785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 613.353575] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 616.157434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 616.281371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 617.758372] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 620.171831] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 621.906783] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 624.051325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 624.173726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 626.072385] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 628.333899] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 636.451242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 641.505376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 641.506842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 641.508463] Lustre: Skipped 2 previous similar messages [ 641.511665] Lustre: Skipped 2 previous similar messages [ 643.943036] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.59@tcp (stopping) [ 646.625136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 646.627637] Lustre: Skipped 2 previous similar messages [ 647.190188] Lustre: server umount lustre-MDT0000 complete [ 648.395446] LustreError: 39900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786542285 with bad export cookie 5143167076913057401 [ 648.399190] LustreError: 39900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 648.399484] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 649.054506] LustreError: 39915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 649.060845] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.59@tcp (stopping) [ 649.061583] LustreError: 39915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 651.743775] LustreError: 39916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 651.744097] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 653.791825] LustreError: 39914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 653.792219] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 653.799145] Lustre: Skipped 1 previous similar message [ 654.579562] Lustre: server umount lustre-MDT0001 complete [ 672.466327] Lustre: server umount lustre-OST0000 complete [ 677.512719] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 678.552203] Key type lgssc unregistered [ 678.677321] LNet: 43802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 678.679437] LNetError: 43802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 678.692287] LNet: Removed LNI 192.168.203.159@tcp [ 678.969619] Key type .llcrypt unregistered [ 678.970629] Key type ._llcrypt unregistered [ 687.173671] Key type ._llcrypt registered [ 687.174611] Key type .llcrypt registered [ 687.216901] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 687.654504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 687.694828] alg: No test for adler32 (adler32-zlib) [ 688.557634] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 688.646275] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 690.223197] Key type lgssc registered [ 690.581974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 693.021614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 693.027333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 694.157528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 695.370406] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 698.009913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.121742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 699.389769] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 701.539553] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 703.202297] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 705.334868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 705.452272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 707.207976] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 709.451818] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 719.781951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 728.031680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 728.034134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 728.037823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 729.957190] Lustre: lustre-OST0000: Not available for connect from 192.168.203.59@tcp (stopping) [ 730.081630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 730.081805] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 733.006943] Lustre: server umount lustre-OST0000 complete [ 749.536071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 749.539792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 749.544205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 749.546291] Lustre: Skipped 2 previous similar messages [ 751.827221] Lustre: server umount lustre-MDT0000 complete [ 752.945055] LustreError: 47035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786542389 with bad export cookie 137382394116684901 [ 752.949028] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 752.949612] LustreError: 47035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 759.167792] Lustre: server umount lustre-MDT0001 complete [ 762.592185] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 763.583700] Key type lgssc unregistered [ 763.704496] LNet: 48923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.707848] LNetError: 48923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 763.716327] LNet: Removed LNI 192.168.203.159@tcp [ 764.019782] Key type .llcrypt unregistered [ 764.020964] Key type ._llcrypt unregistered [ 772.294223] Key type ._llcrypt registered [ 772.295222] Key type .llcrypt registered [ 772.327311] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 772.703770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 772.709561] alg: No test for adler32 (adler32-zlib) [ 773.555054] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 773.640683] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 775.215091] Key type lgssc registered [ 775.533721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 777.934236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 777.939892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 779.071216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 780.270501] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 782.913600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 783.033956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 784.412666] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 786.711523] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 788.480762] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 790.640344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 790.772944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 792.505365] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 794.857671] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 805.094330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 807.183780] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 808.219129] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 818.860399] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-12 9h47m35s [ 819.305591] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 820.580522] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.59@tcp (stopping) [ 820.705642] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 820.708959] Lustre: Skipped 2 previous similar messages [ 821.598409] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.59@tcp (stopping) [ 821.601314] Lustre: Skipped 3 previous similar messages [ 826.134244] Lustre: server umount lustre-MDT0000 complete [ 827.288097] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 827.742211] LustreError: 50340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 827.749458] LustreError: 50340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 829.332600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 829.371128] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 829.462252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 829.489590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 830.501598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 830.740368] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 831.758594] Lustre: DEBUG MARKER: Wait for df (36547) ... [ 832.240891] Lustre: DEBUG MARKER: done [ 834.015750] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 834.018019] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 834.021725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 834.023708] Lustre: Skipped 3 previous similar messages [ 834.528245] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 835.214360] Lustre: server umount lustre-OST0000 complete [ 840.672475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 840.673074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 840.673281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 840.673286] Lustre: Skipped 1 previous similar message [ 840.676537] Lustre: Skipped 2 previous similar messages [ 842.519663] Lustre: server umount lustre-MDT0000 complete [ 843.732066] LustreError: 52962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786542480 with bad export cookie 13734322316008350723 [ 843.735370] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 843.736390] LustreError: 52962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 843.840580] Lustre: server umount lustre-MDT0001 complete [ 847.145212] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 848.173490] Key type lgssc unregistered [ 848.301444] LNet: 55251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.304236] LNetError: 55251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.313296] LNet: Removed LNI 192.168.203.159@tcp [ 848.631152] Key type .llcrypt unregistered [ 848.632165] Key type ._llcrypt unregistered [ 861.843832] Key type ._llcrypt registered [ 861.845662] Key type .llcrypt registered [ 861.886852] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 862.183782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 862.214317] alg: No test for adler32 (adler32-zlib) [ 863.057585] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 863.131207] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 864.703138] Key type lgssc registered [ 865.080189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 867.705256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 867.712660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 868.803124] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 868.810091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 868.867357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 870.137300] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 872.759394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 872.789911] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 872.800617] Lustre: 57436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 872.888845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 874.181727] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 876.477328] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 878.152091] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 880.269803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 880.339562] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 880.399671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 882.190400] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 884.359602] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 885.254703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 892.807702] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-12 9h48m49s [ 893.304281] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 895.680281] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 896.469751] Lustre: DEBUG MARKER: Creating a test file and stat it [ 896.483422] LustreError: 57442:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 916.839785] Lustre: lustre-MDT0000: Client 2e2d3f95-ba9a-404e-932b-5a8e457480f8 (at 192.168.203.59@tcp) reconnecting [ 936.527263] LustreError: 57442:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 937.225783] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 938.024945] Lustre: DEBUG MARKER: done [ 941.536809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 941.538094] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 941.539223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 941.543500] Lustre: Skipped 1 previous similar message [ 945.618216] Lustre: server umount lustre-OST0000 complete [ 949.727848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 949.731149] LustreError: Skipped 1 previous similar message [ 949.733304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 949.739350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 949.741530] Lustre: Skipped 1 previous similar message [ 950.752316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 950.753846] Lustre: Skipped 2 previous similar messages [ 952.800606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 952.802357] Lustre: Skipped 1 previous similar message [ 952.923956] Lustre: server umount lustre-MDT0000 complete [ 954.142577] LustreError: 57426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786542590 with bad export cookie 2017618057203376241 [ 954.145485] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 954.146182] LustreError: 57426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 954.274310] Lustre: server umount lustre-MDT0001 complete [ 957.685651] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 958.642386] Key type lgssc unregistered [ 958.763345] LNet: 61762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.766286] LNetError: 61762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.776279] LNet: Removed LNI 192.168.203.159@tcp [ 959.061060] Key type .llcrypt unregistered [ 959.062234] Key type ._llcrypt unregistered [ 973.401361] Key type ._llcrypt registered [ 973.402726] Key type .llcrypt registered [ 973.440234] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 973.905594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 973.931284] alg: No test for adler32 (adler32-zlib) [ 974.792131] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 974.868921] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 976.447128] Key type lgssc registered [ 976.844548] Lustre: Echo OBD driver; http://www.lustre.org/ [ 977.953144] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 977.959910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 979.050750] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 979.058173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 979.114700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 979.458356] Lustre: Failing over lustre-MDT0000 [ 979.554491] Lustre: server umount lustre-MDT0000 complete [ 981.771741] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:50:18 (1786542618) [ 983.945493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 984.097644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 985.465053] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 988.388817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 988.416341] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 988.426383] Lustre: 64537:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 988.509036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 989.748634] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 991.978202] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 993.650689] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 995.637529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 995.706043] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 995.762050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 997.394582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 997.693378] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1000.127984] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1003.139594] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1007.586063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1007.586811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1007.589725] Lustre: Skipped 2 previous similar messages [ 1007.591360] Lustre: Skipped 1 previous similar message [ 1010.978205] Lustre: server umount lustre-MDT0000 complete [ 1012.178238] LustreError: 64529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786542648 with bad export cookie 7166731092245042271 [ 1012.181603] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1012.185987] LustreError: 64529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1012.293653] Lustre: server umount lustre-MDT0001 complete [ 1013.331367] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1014.825734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1017.535678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1021.855597] LustreError: 63461:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008e29156f x1873325704902656/t0(0) o250->MGC192.168.203.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1022.006554] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1022.009248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1022.035630] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1023.356889] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1026.194435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1027.364020] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1027.644883] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1029.151129] Lustre: 63463:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786542649/real 1786542649] req@00000000d721d771 x1873325704902400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786542665 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1029.968014] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1031.795958] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1033.679110] Lustre: 63464:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786542654/real 1786542654] req@00000000c8180a12 x1873325704902528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786542670 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1036.768154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1036.771845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1036.776783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1036.780145] Lustre: Skipped 1 previous similar message [ 1037.792104] Lustre: 63464:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786542658/real 1786542658] req@00000000b7ffd3ee x1873325704902784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786542674 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1037.794391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1037.806922] Lustre: Skipped 2 previous similar messages [ 1039.704826] Lustre: server umount lustre-MDT0000 complete [ 1040.940115] LustreError: 64568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786542677 with bad export cookie 7166731092245044539 [ 1040.943251] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1040.943733] LustreError: 64568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1041.048154] Lustre: server umount lustre-MDT0001 complete [ 1043.161032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1046.348609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1051.040072] LustreError: 63461:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cd57f79f x1873325704922880/t0(0) o250->MGC192.168.203.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1051.168654] LustreError: 66385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1051.175336] LustreError: 66385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1051.195911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1051.197799] Lustre: Skipped 1 previous similar message [ 1051.220930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1052.520743] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1055.265043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1055.357411] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1055.363098] Lustre: Skipped 2 previous similar messages [ 1056.778749] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1057.444891] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1057.447159] Lustre: Skipped 1 previous similar message [ 1058.463195] Lustre: 63463:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786542679/real 1786542679] req@0000000044fa41ee x1873325704922496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786542695 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1059.249949] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1061.024952] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1062.881198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1062.882666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1062.887621] Lustre: Skipped 2 previous similar messages [ 1062.891850] Lustre: Skipped 2 previous similar messages [ 1063.903099] Lustre: 63464:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786542684/real 1786542684] req@0000000000f5c53a x1873325704922752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786542700 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1063.910303] Lustre: 63464:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1068.001785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1068.003911] Lustre: Skipped 2 previous similar messages [ 1068.886736] Lustre: server umount lustre-MDT0000 complete [ 1070.141209] LustreError: 65272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786542706 with bad export cookie 7166731092245045918 [ 1070.144905] LustreError: 65272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1070.144967] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1070.260086] Lustre: server umount lustre-MDT0001 complete [ 1072.428960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1073.550826] Lustre: DEBUG MARKER: files compared the same [ 1083.488662] Lustre: server umount lustre-OST0000 complete [ 1089.209639] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1090.267699] Key type lgssc unregistered [ 1090.404438] LNet: 74250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1090.406849] LNetError: 74250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1090.418299] LNet: Removed LNI 192.168.203.159@tcp [ 1090.722810] Key type .llcrypt unregistered [ 1090.723987] Key type ._llcrypt unregistered [ 1100.050563] Key type ._llcrypt registered [ 1100.051858] Key type .llcrypt registered [ 1100.092535] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1100.433886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1100.441680] alg: No test for adler32 (adler32-zlib) [ 1101.290383] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 1101.377767] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1102.959181] Key type lgssc registered [ 1103.316235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1105.933310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1105.942179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1107.092856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.414691] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug +malloc all [ 1111.181731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1111.281701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1112.617810] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug +malloc all [ 1114.992489] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1116.810092] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1119.030715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1119.135209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1121.116538] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug +malloc all [ 1123.601864] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1131.426503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1136.607866] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1136.607876] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1136.608236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1136.610209] LustreError: Skipped 1 previous similar message [ 1136.623510] Lustre: Skipped 1 previous similar message [ 1141.728188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1141.728188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1141.728201] Lustre: Skipped 1 previous similar message [ 1141.731085] Lustre: Skipped 1 previous similar message [ 1142.601990] Lustre: server umount lustre-OST0000 complete [ 1146.847640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.847942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1146.851186] Lustre: Skipped 1 previous similar message [ 1151.967681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1151.971781] Lustre: Skipped 2 previous similar messages [ 1157.087517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1157.089365] Lustre: Skipped 1 previous similar message [ 1158.111145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1158.170628] Lustre: server umount lustre-MDT0000 complete [ 1159.395253] LustreError: 75641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786542796 with bad export cookie 8080006528602354396 [ 1159.397181] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1159.400305] LustreError: 75641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1159.511922] Lustre: server umount lustre-MDT0001 complete [ 1163.153052] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1164.158297] Key type lgssc unregistered [ 1164.289325] LNet: 79215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1164.293143] LNetError: 79215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1164.299277] LNet: Removed LNI 192.168.203.159@tcp [ 1164.621079] Key type .llcrypt unregistered [ 1164.622224] Key type ._llcrypt unregistered [ 1173.718244] Key type ._llcrypt registered [ 1173.719296] Key type .llcrypt registered [ 1173.754350] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1174.069487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1174.078947] alg: No test for adler32 (adler32-zlib) [ 1174.926930] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 1175.024175] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1176.607160] Key type lgssc registered [ 1176.987486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1179.559662] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1179.565536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1194.975402] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1200.095379] LustreError: 80582:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.159@tcp: failed processing log, type 4: rc = -110 [ 1226.719199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1228.614143] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1231.100296] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1234.509658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1234.559296] Lustre: *** cfs_fail_loc=706, val=0*** [ 1234.725983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1234.761140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1236.056286] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1239.155421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1239.307804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1240.917067] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1243.329633] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1245.176875] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1246.690323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1246.691739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1246.694148] Lustre: Skipped 1 previous similar message [ 1246.697156] Lustre: Skipped 1 previous similar message [ 1251.810802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1251.812487] Lustre: Skipped 1 previous similar message [ 1251.981413] Lustre: server umount lustre-OST0000 complete [ 1254.879905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1254.882480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1254.886449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1256.928259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1256.929227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1259.349699] Lustre: server umount lustre-MDT0000 complete [ 1260.612184] LustreError: 80590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786542897 with bad export cookie 4311072945313095843 [ 1260.615518] LustreError: 80590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1260.615650] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1260.731357] Lustre: server umount lustre-MDT0001 complete [ 1264.039302] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1264.993751] Key type lgssc unregistered [ 1265.107360] LNet: 84195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1265.110096] LNetError: 84195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1265.119287] LNet: Removed LNI 192.168.203.159@tcp [ 1265.391670] Key type .llcrypt unregistered [ 1265.392831] Key type ._llcrypt unregistered [ 1274.389800] Key type ._llcrypt registered [ 1274.390865] Key type .llcrypt registered [ 1274.425675] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1274.733867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1274.766475] alg: No test for adler32 (adler32-zlib) [ 1275.617161] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 1275.697854] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1277.271128] Key type lgssc registered [ 1277.680961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1280.397887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1280.404158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1282.724933] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1285.194840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1285.373590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1286.714206] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1289.508273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1289.619428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1291.496126] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1294.229863] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1294.236386] Lustre: MGS: Client ffc91092-b24b-4965-8cca-81f7251a24aa (at 0@lo) reconnecting [ 1294.243638] LustreError: 86291:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1294.322326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1294.353253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1295.651323] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1304.544383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1304.546386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1304.549772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1305.570772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1305.573094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1305.574359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1305.586070] Lustre: Skipped 1 previous similar message [ 1309.515791] Lustre: server umount lustre-OST0000 complete [ 1315.808645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1315.809414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1315.809554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1315.815024] Lustre: Skipped 1 previous similar message [ 1317.066202] Lustre: server umount lustre-MDT0000 complete [ 1318.602464] Lustre: server umount lustre-MDT0001 complete [ 1319.999097] Lustre: server umount MGS complete [ 1323.695095] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:56:00 (1786542960) [ 1332.664196] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 1335.745941] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1347.191212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.610306] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1352.367342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1352.456915] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1352.474720] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1352.517439] Lustre: lustre-MDT0001: new disk, initializing [ 1352.540860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1352.549502] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1352.551792] Lustre: 93433:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1352.554385] LustreError: 93433:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1353.819982] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1357.883074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1357.988228] Lustre: lustre-OST0000: new disk, initializing [ 1357.989835] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1357.992236] Lustre: 94199:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1358.011662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1359.845587] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1363.408510] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1365.023961] Lustre: 94963:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000021567f53 x1873326020527488/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786543017 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1365.645519] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1365.650165] Lustre: MGS: Client e39eaac3-ea41-44aa-afe7-23c71ff89881 (at 0@lo) reconnecting [ 1365.665068] Lustre: 95244:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1365.674856] LustreError: 93418:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1365.677688] LustreError: 93418:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1365.683719] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1365.725840] Lustre: lustre-MDT0000: new disk, initializing [ 1365.783098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1366.816869] Lustre: 95268:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000005d95fb89 x1873326020532736/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786543019 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1367.072719] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1367.181550] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1371.874681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1371.877845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1371.944857] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1391.584663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1391.587721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1391.588262] Lustre: Skipped 1 previous similar message [ 1391.590669] Lustre: Skipped 1 previous similar message [ 1396.708780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1396.712161] Lustre: Skipped 1 previous similar message [ 1397.335899] Lustre: server umount lustre-OST0000 complete [ 1401.826274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1401.827214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1401.836243] Lustre: Skipped 1 previous similar message [ 1404.808725] Lustre: server umount lustre-MDT0000 complete [ 1412.198224] Lustre: server umount lustre-MDT0001 complete [ 1413.491918] Lustre: server umount MGS complete [ 1417.185133] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:57:33 (1786543053) [ 1423.210714] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1424.248642] Key type lgssc unregistered [ 1424.371416] LNet: 97765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1424.374704] LNetError: 97765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1424.382315] LNet: Removed LNI 192.168.203.159@tcp [ 1424.722932] Key type .llcrypt unregistered [ 1424.724226] Key type ._llcrypt unregistered [ 1431.895147] Key type ._llcrypt registered [ 1431.896202] Key type .llcrypt registered [ 1431.932957] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1432.441606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1432.451098] alg: No test for adler32 (adler32-zlib) [ 1433.304528] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 1433.387894] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1434.967098] Key type lgssc registered [ 1435.324401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1438.043142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1438.049483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1439.200447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1440.553465] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1443.395596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1443.509152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1444.845108] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1447.376988] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1449.387960] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1451.690426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1451.809883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1453.881535] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1456.272563] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1464.107810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1474.532205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1474.532475] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1474.538409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1474.541479] LustreError: Skipped 1 previous similar message [ 1474.545875] Lustre: Skipped 1 previous similar message [ 1475.536559] Lustre: server umount lustre-OST0000 complete [ 1479.649261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1479.649837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1479.653151] Lustre: Skipped 2 previous similar messages [ 1479.653384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1479.654972] Lustre: Skipped 1 previous similar message [ 1482.903729] Lustre: server umount lustre-MDT0000 complete [ 1484.117198] LustreError: 98946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543120 with bad export cookie 6858540477535322805 [ 1484.120609] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1484.122143] LustreError: 98946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1484.230683] Lustre: server umount lustre-MDT0001 complete [ 1488.018835] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1488.966678] Key type lgssc unregistered [ 1489.085365] LNet: 102517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1489.088139] LNetError: 102517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1489.099293] LNet: Removed LNI 192.168.203.159@tcp [ 1489.418247] Key type .llcrypt unregistered [ 1489.419393] Key type ._llcrypt unregistered [ 1496.195477] Key type ._llcrypt registered [ 1496.196523] Key type .llcrypt registered [ 1496.229588] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1496.563603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1496.603543] alg: No test for adler32 (adler32-zlib) [ 1497.448237] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 1497.542767] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1499.119134] Key type lgssc registered [ 1499.507573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1501.577804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1501.577834] LustreError: 103576:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1501.580537] Lustre: Skipped 1 previous similar message [ 1501.584813] LustreError: 103577:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1501.587854] LustreError: 103576:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 1501.590576] LustreError: 103576:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1501.593808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1501.594749] LustreError: 103576:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1502.734590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1505.336586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1505.442793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1506.795652] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1508.543648] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing lsmod [ 1509.504668] LustreError: 104782:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1509.509591] LustreError: 104783:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1509.512399] LustreError: 104782:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 1509.514641] LustreError: 104782:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1509.518482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1509.521031] LustreError: 104782:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1509.662346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1510.880109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1510.883733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1510.889638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1514.977027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1514.978330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1514.980310] Lustre: Skipped 1 previous similar message [ 1514.984702] Lustre: Skipped 1 previous similar message [ 1516.002085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1516.004917] Lustre: Skipped 2 previous similar messages [ 1516.759950] Lustre: server umount lustre-MDT0000 complete [ 1518.000255] LustreError: 103588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543154 with bad export cookie 5868321163079492636 [ 1518.002721] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1518.006777] LustreError: 103588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1518.047184] LustreError: 104046:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001175541a x1873326253293056/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1518.119388] Lustre: server umount lustre-MDT0001 complete [ 1529.366062] Lustre: server umount lustre-OST0000 complete [ 1532.062415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1532.213366] LustreError: 105844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1532.219586] LustreError: 105844:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1532.236387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1533.567566] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1536.411734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1536.543664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1537.928801] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1540.347777] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1542.155685] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1544.384667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1546.381277] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1548.737249] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1557.807319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1562.081529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1562.083755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1562.088118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1563.108597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1566.922972] Lustre: server umount lustre-OST0000 complete [ 1568.224548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1568.225293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1568.230018] Lustre: Skipped 2 previous similar messages [ 1568.232410] Lustre: Skipped 2 previous similar messages [ 1574.296367] Lustre: server umount lustre-MDT0000 complete [ 1575.471070] LustreError: 105824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543212 with bad export cookie 5868321163079493301 [ 1575.475827] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1575.476859] LustreError: 105824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1575.587862] Lustre: server umount lustre-MDT0001 complete [ 1579.230568] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1580.474725] Key type lgssc unregistered [ 1580.597237] LNet: 109396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1580.599622] LNetError: 109396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1580.611281] LNet: Removed LNI 192.168.203.159@tcp [ 1580.908642] Key type .llcrypt unregistered [ 1580.909891] Key type ._llcrypt unregistered [ 1589.807467] Key type ._llcrypt registered [ 1589.808486] Key type .llcrypt registered [ 1589.845503] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1590.259442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1590.266305] alg: No test for adler32 (adler32-zlib) [ 1591.122968] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 1591.209865] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1592.791170] Key type lgssc registered [ 1593.126556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1595.606917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1595.612829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1596.748416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1598.093388] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1600.826432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1600.948189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1602.219216] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1604.523833] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1606.286465] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1608.466610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1608.579339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1610.453141] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1613.217209] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1621.926304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1625.856316] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1626.600548] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1631.199856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1631.202956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1631.207518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1632.225792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1632.226889] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1632.227382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1632.228906] Lustre: Skipped 1 previous similar message [ 1633.679557] Lustre: server umount lustre-OST0000 complete [ 1636.832050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1636.834667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1636.839285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1641.059992] Lustre: server umount lustre-MDT0000 complete [ 1642.278954] LustreError: 110771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543278 with bad export cookie 14584159855673965087 [ 1642.282214] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1642.282470] LustreError: 110771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1642.394080] Lustre: server umount lustre-MDT0001 complete [ 1646.019234] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1647.030608] Key type lgssc unregistered [ 1647.163388] LNet: 114460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1647.168062] LNetError: 114460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1647.178332] LNet: Removed LNI 192.168.203.159@tcp [ 1647.497576] Key type .llcrypt unregistered [ 1647.498924] Key type ._llcrypt unregistered [ 1654.743279] Key type ._llcrypt registered [ 1654.744387] Key type .llcrypt registered [ 1654.782308] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1655.156807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1655.176496] alg: No test for adler32 (adler32-zlib) [ 1656.037982] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 1656.132297] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1657.719278] Key type lgssc registered [ 1658.199937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1660.738641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1660.745972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1661.890985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1663.201936] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1666.251754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1666.378612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1667.929539] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1671.034681] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1672.833640] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1675.261878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1675.378466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1677.379964] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1679.819843] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1686.630600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 1697.249965] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1697.252050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1697.253718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1697.257449] Lustre: Skipped 1 previous similar message [ 1699.210808] Lustre: server umount lustre-OST0000 complete [ 1702.368555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1702.369129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1702.370984] LustreError: Skipped 1 previous similar message [ 1702.376633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1702.378444] Lustre: Skipped 2 previous similar messages [ 1706.593782] Lustre: server umount lustre-MDT0000 complete [ 1707.488409] LustreError: 116387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1707.496369] LustreError: 116387:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1708.058577] LustreError: 117498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543344 with bad export cookie 3903172496473442487 [ 1708.064643] LustreError: 117498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1708.066505] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1708.223538] Lustre: server umount lustre-MDT0001 complete [ 1711.921736] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1713.027436] Key type lgssc unregistered [ 1713.140551] LNet: 119210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1713.145602] LNetError: 119210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1713.155426] LNet: Removed LNI 192.168.203.159@tcp [ 1713.505512] Key type .llcrypt unregistered [ 1713.506545] Key type ._llcrypt unregistered [ 1723.678745] Key type ._llcrypt registered [ 1723.680724] Key type .llcrypt registered [ 1723.736921] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1724.271605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1724.311558] alg: No test for adler32 (adler32-zlib) [ 1725.172513] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 1725.263438] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1726.839271] Key type lgssc registered [ 1727.283197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1730.246431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1730.253220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1731.418469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1733.058880] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1737.099253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1737.292444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1739.155331] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1742.135976] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1744.518536] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1747.667107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1747.803191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1750.471642] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1753.302243] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1757.043778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 1765.836454] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1767.106454] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1767.108842] Lustre: Skipped 2 previous similar messages [ 1768.628781] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1768.633494] Lustre: Skipped 2 previous similar messages [ 1775.121165] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1775.140795] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1778.186820] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.59@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1778.194099] Lustre: Skipped 2 previous similar messages [ 1785.337913] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.59@tcp 192.168.0.[2,10]@tcp [ 1788.212652] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.59/24@tcp in log lustre-MDT0000 [ 1788.215222] Lustre: Skipped 2 previous similar messages [ 1792.267915] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.59/24@tcp [ 1792.273372] Lustre: Skipped 1 previous similar message [ 1798.112568] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1798.112772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1798.114961] LustreError: Skipped 1 previous similar message [ 1798.119210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1798.121940] Lustre: Skipped 1 previous similar message [ 1801.186075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1801.189319] Lustre: Skipped 1 previous similar message [ 1802.472603] Lustre: server umount lustre-OST0000 complete [ 1806.306890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1806.307781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1806.319384] Lustre: Skipped 2 previous similar messages [ 1806.327249] Lustre: Skipped 1 previous similar message [ 1810.473306] Lustre: server umount lustre-MDT0000 complete [ 1811.425557] LustreError: 120603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1811.430419] LustreError: 120603:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1811.698457] LustreError: 120584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543448 with bad export cookie 13801232313079661628 [ 1811.702561] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1811.703944] LustreError: 120584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1811.827426] Lustre: server umount lustre-MDT0001 complete [ 1815.676602] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1816.718456] Key type lgssc unregistered [ 1816.852440] LNet: 125844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1816.856265] LNetError: 125844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1816.864359] LNet: Removed LNI 192.168.203.159@tcp [ 1817.189943] Key type .llcrypt unregistered [ 1817.191346] Key type ._llcrypt unregistered [ 1828.986207] Key type ._llcrypt registered [ 1828.987720] Key type .llcrypt registered [ 1829.022724] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1829.438219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1829.452653] alg: No test for adler32 (adler32-zlib) [ 1830.315735] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 1830.414683] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1831.999217] Key type lgssc registered [ 1832.392789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1835.440250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1835.447313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1836.618543] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1836.620376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1836.629227] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.59/24@tcp [ 1838.282619] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1841.387736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1841.500412] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1841.509346] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.59/24@tcp [ 1841.524116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1842.918219] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1845.560358] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1847.402122] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1849.888715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1850.011652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1851.906459] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1854.743537] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1862.313901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 1872.865959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1872.866616] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1872.866879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1872.869682] Lustre: Skipped 1 previous similar message [ 1874.765168] Lustre: server umount lustre-OST0000 complete [ 1877.471834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1877.474267] LustreError: Skipped 1 previous similar message [ 1877.475743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1877.480082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1877.482845] Lustre: Skipped 1 previous similar message [ 1882.273863] Lustre: server umount lustre-MDT0000 complete [ 1883.104728] LustreError: 127400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1883.109433] LustreError: 127400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1883.555077] LustreError: 128122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543520 with bad export cookie 6680130185581809070 [ 1883.558755] LustreError: 128122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1883.559750] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1883.696848] Lustre: server umount lustre-MDT0001 complete [ 1887.432097] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1888.701791] Key type lgssc unregistered [ 1888.887520] LNet: 130990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1888.891768] LNetError: 130990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1888.902320] LNet: Removed LNI 192.168.203.159@tcp [ 1889.285199] Key type .llcrypt unregistered [ 1889.287118] Key type ._llcrypt unregistered [ 1898.598662] Key type ._llcrypt registered [ 1898.599804] Key type .llcrypt registered [ 1898.640503] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 1905.280655] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1905.635221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1905.682767] alg: No test for adler32 (adler32-zlib) [ 1906.531710] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 1906.614889] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1908.191210] Key type lgssc registered [ 1908.595060] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1920.515660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1920.521599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1921.618711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1921.630612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1921.674696] Lustre: lustre-MDT0000: new disk, initializing [ 1921.700207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1921.708405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1923.193076] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1928.176270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1928.234113] Lustre: 134364:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1928.277782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1928.280160] Lustre: Skipped 1 previous similar message [ 1928.318443] Lustre: lustre-MDT0001: new disk, initializing [ 1928.365948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1928.387702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1928.392246] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1929.861796] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1934.075502] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1935.944567] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1938.293598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1938.405658] Lustre: lustre-OST0000: new disk, initializing [ 1938.407724] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1938.410491] Lustre: 136333:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1938.448486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1940.684076] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1944.762682] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1946.563734] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1948.673280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1948.678896] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1948.723830] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1949.770666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 1951.516943] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1951.615368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1953.767327] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1953.769343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.771224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1953.773223] LustreError: Skipped 1 previous similar message [ 1953.781397] Lustre: Skipped 1 previous similar message [ 1958.611239] Lustre: server umount lustre-OST0000 complete [ 1964.002218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.003439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1964.010379] Lustre: Skipped 1 previous similar message [ 1964.017714] Lustre: Skipped 2 previous similar messages [ 1966.258836] Lustre: server umount lustre-MDT0000 complete [ 1967.722277] LustreError: 134401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543604 with bad export cookie 18021653573179189825 [ 1967.728458] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1967.730400] LustreError: 134401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1967.877908] Lustre: server umount lustre-MDT0001 complete [ 1971.148279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.368564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1972.837945] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1976.227870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1976.371675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1977.933548] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1981.027377] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1983.086103] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1985.736877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1985.845830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1987.850402] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1990.555156] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1992.058010] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1998.299301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 1999.844769] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1999.922336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2003.532384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2003.603857] Lustre: lustre-OST0001: new disk, initializing [ 2003.605539] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2003.607859] Lustre: 141981:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2003.637253] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2003.707968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2003.711113] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2003.753902] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2005.635786] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2009.172982] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2009.239484] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2010.626059] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2010.696685] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2012.492828] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9574a08c4000.ost_server_uuid 50 [ 2013.001984] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9574a08c4000.ost_server_uuid in FULL state after 0 sec [ 2019.299662] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2019.304826] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2019.305264] LustreError: Skipped 1 previous similar message [ 2019.315428] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2019.322107] Lustre: Skipped 1 previous similar message [ 2020.884334] Lustre: server umount lustre-OST0001 complete [ 2022.879832] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2022.884030] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2022.890161] Lustre: Skipped 1 previous similar message [ 2022.892904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2024.423076] LustreError: 140392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2024.433514] LustreError: 140392:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2028.306278] Lustre: server umount lustre-OST0000 complete [ 2032.607991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2032.612661] LustreError: Skipped 1 previous similar message [ 2032.614382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2032.620807] Lustre: Skipped 1 previous similar message [ 2032.623445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2032.627016] Lustre: Skipped 3 previous similar messages [ 2035.680785] Lustre: server umount lustre-MDT0000 complete [ 2036.945226] LustreError: 141979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543673 with bad export cookie 18021653573179190693 [ 2036.948938] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2036.952745] LustreError: 141979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2037.073816] Lustre: server umount lustre-MDT0001 complete [ 2043.534803] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 2044.685977] Key type lgssc unregistered [ 2044.828484] LNet: 144982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2044.831958] LNetError: 144982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2044.839407] LNet: Removed LNI 192.168.203.159@tcp [ 2045.183721] Key type .llcrypt unregistered [ 2045.185145] Key type ._llcrypt unregistered [ 2059.335661] Key type ._llcrypt registered [ 2059.336905] Key type .llcrypt registered [ 2059.370757] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 2066.353854] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2066.919782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2066.930665] alg: No test for adler32 (adler32-zlib) [ 2067.798326] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 2067.898887] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 2069.487101] Key type lgssc registered [ 2069.919156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2082.868389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2082.874289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2083.963298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2083.973227] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2084.013090] Lustre: lustre-MDT0000: new disk, initializing [ 2084.034906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2084.042040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2085.661205] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2089.958125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2089.999941] Lustre: 149115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2090.023951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2090.026650] Lustre: Skipped 1 previous similar message [ 2090.071494] Lustre: lustre-MDT0001: new disk, initializing [ 2090.108482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2090.120503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2090.124177] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2091.458076] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2095.585978] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2097.443125] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2099.994529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2100.091967] Lustre: lustre-OST0000: new disk, initializing [ 2100.093582] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2100.096248] Lustre: 151082:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2100.121905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2101.987461] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2105.722352] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2108.411291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2108.415284] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2108.447164] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2113.128851] Lustre: Failing over lustre-OST0000 [ 2113.507531] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2113.507883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2113.508997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2113.512892] LustreError: Skipped 1 previous similar message [ 2113.518249] Lustre: Skipped 1 previous similar message [ 2115.213459] Lustre: server umount lustre-OST0000 complete [ 2128.344219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2128.485444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2128.493771] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2130.363776] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2131.561021] Lustre: Failing over lustre-MDT0000 [ 2131.567641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.568157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2131.572254] Lustre: Skipped 1 previous similar message [ 2131.574166] Lustre: Skipped 1 previous similar message [ 2131.662411] Lustre: server umount lustre-MDT0000 complete [ 2132.834565] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2133.665765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2137.951077] LustreError: 150522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2137.957919] LustreError: 150522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2139.105283] LustreError: 149127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.115202] LustreError: 149127:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2143.070677] LustreError: 149123:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2144.624301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2144.662348] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2144.666488] Lustre: 149120:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.159@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2144.738939] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2144.756464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2144.783155] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2146.025407] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2148.192062] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2149.862927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2149.871888] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2149.876228] Lustre: Skipped 1 previous similar message [ 2149.897511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2154.977311] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2154.978156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2154.979677] LustreError: Skipped 1 previous similar message [ 2154.981166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2154.981171] Lustre: Skipped 1 previous similar message [ 2154.983152] Lustre: Skipped 1 previous similar message [ 2157.081708] Lustre: server umount lustre-OST0000 complete [ 2160.097197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2160.097615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2160.107494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2160.110279] Lustre: Skipped 2 previous similar messages [ 2164.445938] Lustre: server umount lustre-MDT0000 complete [ 2165.216098] LustreError: 149128:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2165.222214] LustreError: 149128:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2165.827937] LustreError: 149107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543802 with bad export cookie 14042297550938276254 [ 2165.831109] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2165.834327] LustreError: 149107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2165.978188] Lustre: server umount lustre-MDT0001 complete [ 2170.116287] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 2171.252849] Key type lgssc unregistered [ 2171.381447] LNet: 154732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2171.384223] LNetError: 154732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2171.392399] LNet: Removed LNI 192.168.203.159@tcp [ 2171.740276] Key type .llcrypt unregistered [ 2171.741316] Key type ._llcrypt unregistered [ 2192.025217] Key type ._llcrypt registered [ 2192.026485] Key type .llcrypt registered [ 2192.070443] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2192.470814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2192.503075] alg: No test for adler32 (adler32-zlib) [ 2193.382858] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 2193.482806] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 2195.071157] Key type lgssc registered [ 2195.504324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2198.548062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2198.554731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2199.730219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2201.298478] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2205.018515] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2207.022474] LustreError: 156279:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2207.026807] LustreError: 156279:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2207.030635] LustreError: 156279:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2207.180340] Lustre: server umount lustre-MDT0000 complete [ 2212.312853] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 2215.338140] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2227.956939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2228.045161] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2228.055378] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2228.092592] Lustre: lustre-MDT0000: new disk, initializing [ 2228.115176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2228.122116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2229.505207] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2233.794590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2233.834764] Lustre: 159576:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2233.859618] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2233.861622] Lustre: Skipped 1 previous similar message [ 2233.904013] Lustre: lustre-MDT0001: new disk, initializing [ 2233.938364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2233.952087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2233.956617] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2235.366174] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2239.179834] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2240.995774] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2243.575880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2243.687114] Lustre: lustre-OST0000: new disk, initializing [ 2243.689123] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2243.693617] Lustre: 161545:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2243.721320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2245.691900] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2249.405678] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2249.716949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2249.719973] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2249.752156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2256.387459] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2256.950288] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:11:33 (1786543893) [ 2331.103597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2331.108261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2331.113935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2331.616038] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2331.620413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2331.622368] Lustre: Skipped 1 previous similar message [ 2333.693301] Lustre: server umount lustre-MDT0000 complete [ 2335.217888] LustreError: 159570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786543971 with bad export cookie 18424010737397119524 [ 2335.220119] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2335.227126] LustreError: 159570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2336.735455] LustreError: 159583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2336.735529] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2336.745043] LustreError: 159583:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2336.749298] Lustre: Skipped 1 previous similar message [ 2336.749543] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2340.767385] LustreError: 159584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.767390] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2341.435193] Lustre: server umount lustre-MDT0001 complete [ 2357.215098] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2357.272503] Lustre: server umount lustre-OST0000 complete [ 2360.584607] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 2363.896630] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2377.969389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2378.054232] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2378.068445] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2378.114245] Lustre: lustre-MDT0000: new disk, initializing [ 2378.140640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2378.149047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2379.741773] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2384.737644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2384.784314] Lustre: 166170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2384.813731] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2384.816146] Lustre: Skipped 1 previous similar message [ 2384.860795] Lustre: lustre-MDT0001: new disk, initializing [ 2384.920033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2384.924239] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2386.367692] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2390.289314] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2392.297647] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2394.611606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2394.730260] Lustre: lustre-OST0000: new disk, initializing [ 2394.732545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2394.736215] Lustre: 168138:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2394.771903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2394.773915] Lustre: Skipped 1 previous similar message [ 2395.575985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2395.582067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2395.615552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2396.790287] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2400.812151] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2402.327071] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2402.393153] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2403.801879] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2403.876938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2405.857570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2405.858498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2405.861526] LustreError: Skipped 1 previous similar message [ 2405.865828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2405.868146] Lustre: Skipped 1 previous similar message [ 2410.578572] Lustre: server umount lustre-OST0000 complete [ 2415.589566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2415.596206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.605782] Lustre: Skipped 1 previous similar message [ 2415.609144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2418.301544] Lustre: server umount lustre-MDT0000 complete [ 2420.148980] LustreError: 166165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544056 with bad export cookie 18424010737397278354 [ 2420.150308] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2420.154541] LustreError: 166165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2420.351748] Lustre: server umount lustre-MDT0001 complete [ 2424.320180] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:14:20 (1786544060) [ 2432.401420] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 2435.537167] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2446.533266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2446.619672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2446.630717] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2446.666289] Lustre: lustre-MDT0000: new disk, initializing [ 2446.687049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2446.692699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2448.008581] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2452.267963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2452.322897] Lustre: 172885:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2452.435517] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2453.777857] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2457.712225] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2459.532592] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2461.724777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2461.826692] Lustre: 174860:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2463.785670] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2467.320403] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2467.357758] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2467.453625] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2472.418215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2472.418822] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2472.419493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2472.419500] Lustre: Skipped 2 previous similar messages [ 2472.421648] Lustre: Skipped 2 previous similar messages [ 2472.423372] LustreError: Skipped 1 previous similar message [ 2475.344151] Lustre: server umount lustre-OST0000 complete [ 2482.712194] Lustre: server umount lustre-MDT0000 complete [ 2483.881351] LustreError: 173694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544120 with bad export cookie 18424010737397279222 [ 2483.885236] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2483.886399] LustreError: 173694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2484.003741] Lustre: server umount lustre-MDT0001 complete [ 2486.488529] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:15:23 (1786544123) [ 2491.237491] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 2493.961020] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2505.041747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2505.137883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2505.150716] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2505.153860] Lustre: Skipped 3 previous similar messages [ 2505.195377] Lustre: lustre-MDT0000: new disk, initializing [ 2505.197100] Lustre: Skipped 2 previous similar messages [ 2505.222434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2505.224777] Lustre: Skipped 2 previous similar messages [ 2505.231058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2505.235406] Lustre: Skipped 2 previous similar messages [ 2506.507668] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2510.433063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2510.471439] Lustre: 179172:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2510.561825] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2511.818225] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2515.330448] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2517.009306] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2519.123368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2519.236617] Lustre: 181148:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2521.187409] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2524.814759] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2528.273117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2533.344468] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2533.345203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2533.348280] LustreError: Skipped 1 previous similar message [ 2533.351815] Lustre: Skipped 2 previous similar messages [ 2533.352613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2533.356118] Lustre: Skipped 6 previous similar messages [ 2537.939614] Lustre: server umount lustre-OST0000 complete [ 2541.535944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2545.310870] Lustre: server umount lustre-MDT0000 complete [ 2546.513245] LustreError: 179973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544183 with bad export cookie 18424010737397280307 [ 2546.515816] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2546.517939] LustreError: 179973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2546.636952] Lustre: server umount lustre-MDT0001 complete [ 2550.352568] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 2551.333968] Key type lgssc unregistered [ 2551.459405] LNet: 183008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2551.462790] LNetError: 183008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2551.474319] LNet: Removed LNI 192.168.203.159@tcp [ 2551.794066] Key type .llcrypt unregistered [ 2551.795608] Key type ._llcrypt unregistered [ 2560.904427] Key type ._llcrypt registered [ 2560.905988] Key type .llcrypt registered [ 2560.950263] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2561.340970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2561.360061] alg: No test for adler32 (adler32-zlib) [ 2562.220774] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 2562.312688] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 2563.895150] Key type lgssc registered [ 2564.277292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2566.815494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2566.821437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2567.959347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2569.250426] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2572.052534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2572.180874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2573.606451] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2576.027115] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2577.849646] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2580.111546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2580.237047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2582.194321] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2584.663059] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2592.551416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2608.102280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2608.104124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2608.110992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2608.114327] Lustre: Skipped 1 previous similar message [ 2612.818946] Lustre: server umount lustre-OST0000 complete [ 2618.336927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2618.336992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2618.337594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2618.340915] Lustre: Skipped 2 previous similar messages [ 2618.343289] LustreError: Skipped 1 previous similar message [ 2620.182426] Lustre: server umount lustre-MDT0000 complete [ 2621.451574] LustreError: 186902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544258 with bad export cookie 9687175301614195426 [ 2621.454909] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2621.457941] LustreError: 186902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2621.570739] Lustre: server umount lustre-MDT0001 complete [ 2625.293924] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 2626.296203] Key type lgssc unregistered [ 2626.420367] LNet: 187959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2626.422714] LNetError: 187959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2626.429274] LNet: Removed LNI 192.168.203.159@tcp [ 2626.730451] Key type .llcrypt unregistered [ 2626.731686] Key type ._llcrypt unregistered [ 2635.621553] Key type ._llcrypt registered [ 2635.622757] Key type .llcrypt registered [ 2635.666975] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2636.107843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2636.148102] alg: No test for adler32 (adler32-zlib) [ 2637.002562] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 2637.093984] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 2638.671120] Key type lgssc registered [ 2639.025027] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2641.601996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2641.607984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2642.766558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2644.127963] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2647.031626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2647.180653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2648.591731] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2650.983072] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2652.723261] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2654.955554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2655.065467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2656.914842] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2659.355721] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2667.366612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2672.609995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2672.610598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2672.611049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2672.617278] Lustre: Skipped 1 previous similar message [ 2672.619480] LustreError: Skipped 1 previous similar message [ 2675.046132] Lustre: lustre-OST0000: Not available for connect from 192.168.203.59@tcp (stopping) [ 2675.049587] Lustre: Skipped 1 previous similar message [ 2677.728744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2677.731494] Lustre: Skipped 1 previous similar message [ 2678.220036] Lustre: server umount lustre-OST0000 complete [ 2680.518329] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9574884d9000.ost_server_uuid 50 [ 2681.061242] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2691.109919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2691.110669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2691.113647] Lustre: Skipped 1 previous similar message [ 2696.160538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2696.162963] Lustre: Skipped 2 previous similar messages [ 2697.247053] Lustre: server umount lustre-MDT0000 complete [ 2698.520859] LustreError: 191191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544335 with bad export cookie 8380515025910084304 [ 2698.525448] LustreError: 191191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2698.526933] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2698.670410] Lustre: server umount lustre-MDT0001 complete [ 2701.452286] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:18:58 (1786544338) [ 2703.657188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2703.824556] LustreError: 193261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2703.829390] LustreError: 193261:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2703.855514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2705.224234] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2707.959152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2708.092905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2709.369454] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2711.737733] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2713.494150] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2715.733544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2717.801996] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2720.246376] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2722.626974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2722.734220] Lustre: lustre-OST0001: new disk, initializing [ 2722.736096] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2722.739031] Lustre: 196098:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2722.771965] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2722.774520] Lustre: Skipped 1 previous similar message [ 2724.705303] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2726.846372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2726.851258] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2726.881207] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2728.465116] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2730.338944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2730.341086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2735.420285] Lustre: server umount lustre-OST0000 complete [ 2735.587050] LustreError: 195080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2735.592856] LustreError: 195080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2737.399646] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2737.469216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2738.756257] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2738.830960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2739.040366] LustreError: 195079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.160057] LustreError: 195081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.166180] LustreError: 195081:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2750.946422] LustreError: 196908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2750.949898] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2750.952897] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2750.952904] Lustre: Skipped 2 previous similar messages [ 2750.954986] LustreError: 196908:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2750.958861] Lustre: Skipped 1 previous similar message [ 2755.027460] Lustre: server umount lustre-OST0001 complete [ 2759.647901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2759.650775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.407373] Lustre: server umount lustre-MDT0000 complete [ 2763.840337] LustreError: 196097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544400 with bad export cookie 8380515025910085361 [ 2763.844031] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2763.846812] LustreError: 196097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2763.972400] Lustre: server umount lustre-MDT0001 complete [ 2772.621381] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:20:09 (1786544409) [ 2775.095832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2775.224762] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2775.236864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2775.344097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2776.761240] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2780.431921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2780.484596] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2780.497659] Lustre: 199514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2782.241250] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2784.837927] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2786.939275] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2789.327635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2789.402105] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2791.434863] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2791.838845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2794.187373] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2796.849321] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2796.877063] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2798.140112] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 2799.185321] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2801.914871] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2803.173014] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2803.178581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2803.178591] Lustre: Skipped 4 previous similar messages [ 2803.182481] Lustre: Skipped 2 previous similar messages [ 2808.890148] Lustre: server umount lustre-OST0000 complete [ 2812.255421] LustreError: 201365:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2823.651685] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2823.653458] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2823.656231] LustreError: Skipped 1 previous similar message [ 2823.663366] Lustre: Skipped 1 previous similar message [ 2825.622224] Lustre: server umount lustre-OST0001 complete [ 2833.065559] Lustre: server umount lustre-MDT0000 complete [ 2833.888596] LustreError: 199525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2833.895750] LustreError: 199525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2834.413166] LustreError: 199506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544471 with bad export cookie 8380515025910086523 [ 2834.416960] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2834.417577] LustreError: 199506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2834.533206] Lustre: server umount lustre-MDT0001 complete [ 2842.536882] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:21:19 (1786544479) [ 2848.959376] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 2853.286352] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2870.058734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2870.162733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2870.178363] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2870.230181] Lustre: lustre-MDT0000: new disk, initializing [ 2870.262598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2870.267494] Lustre: Skipped 3 previous similar messages [ 2870.278120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2871.908297] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2876.962386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2877.041554] Lustre: 207580:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2877.093943] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2877.098471] Lustre: Skipped 1 previous similar message [ 2877.148513] Lustre: lustre-MDT0001: new disk, initializing [ 2877.224698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2877.231330] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2878.989110] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2882.827595] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2884.876630] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2887.480731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2887.654269] Lustre: lustre-OST0000: new disk, initializing [ 2887.657953] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2887.663833] Lustre: 209550:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2889.143908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2889.147243] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2889.179858] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2889.800179] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2893.881203] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2895.481330] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2895.576198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2897.066269] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2897.141057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2899.427549] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2899.427572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2899.433453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2899.436803] Lustre: Skipped 3 previous similar messages [ 2899.440737] Lustre: Skipped 9 previous similar messages [ 2912.223133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2912.284864] Lustre: server umount lustre-OST0000 complete [ 2919.806322] Lustre: server umount lustre-MDT0000 complete [ 2919.903978] LustreError: 208745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2919.908276] LustreError: 208745:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2921.136481] LustreError: 207573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544557 with bad export cookie 8380515025910088287 [ 2921.140674] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2921.144156] LustreError: 207573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2921.330459] Lustre: server umount lustre-MDT0001 complete [ 2925.049174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2926.861823] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2930.893362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2932.868335] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2935.938152] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2938.479995] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2941.304863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2941.416322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2941.419931] Lustre: Skipped 4 previous similar messages [ 2943.618517] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2946.336901] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2947.995501] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2950.151211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2951.628209] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2955.818939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 4 sec [ 2959.839957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2959.842302] LustreError: Skipped 1 previous similar message [ 2962.651383] Lustre: server umount lustre-OST0000 complete [ 2964.961859] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2965.069384] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2966.834528] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2991.327806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2994.040847] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2997.455818] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2998.092507] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3005.332174] Lustre: server umount lustre-OST0000 complete [ 3007.968543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3007.971604] LustreError: Skipped 1 previous similar message [ 3007.973113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.977119] Lustre: Skipped 4 previous similar messages [ 3021.279116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3021.354108] Lustre: server umount lustre-MDT0000 complete [ 3021.792147] LustreError: 211761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3021.798536] LustreError: 211761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3022.690952] LustreError: 211743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544659 with bad export cookie 8380515025910089155 [ 3022.693791] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3022.695974] LustreError: 211743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3022.810173] Lustre: server umount lustre-MDT0001 complete [ 3025.746494] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:24:22 (1786544662) [ 3028.513551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3030.182213] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3033.416073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3034.918452] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3037.529517] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3039.378323] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3041.833661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3044.724682] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3047.625692] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3049.206306] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3054.464871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 3056.183468] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3056.257749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3059.279721] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3059.398886] Lustre: lustre-OST0001: new disk, initializing [ 3059.401040] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3059.404368] Lustre: 221185:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3061.380891] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3065.597073] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3066.898895] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3067.911857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3067.915468] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3067.951632] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3068.005359] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3069.477469] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3069.556038] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3072.998351] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3073.006215] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3073.008795] Lustre: Skipped 16 previous similar messages [ 3076.298417] Lustre: server umount lustre-OST0001 complete [ 3078.448616] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3078.547156] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3080.085464] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3080.158172] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3104.642595] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3104.729292] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3104.731696] Lustre: Skipped 5 previous similar messages [ 3107.370723] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3109.867324] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3109.874574] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3109.874802] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3110.617755] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3111.377909] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3114.978544] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3114.983966] LustreError: Skipped 2 previous similar messages [ 3118.403786] Lustre: server umount lustre-OST0001 complete [ 3145.183084] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3145.246900] Lustre: server umount lustre-OST0000 complete [ 3150.819447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3150.826769] Lustre: Skipped 8 previous similar messages [ 3152.737449] Lustre: server umount lustre-MDT0000 complete [ 3154.321864] LustreError: 217806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544791 with bad export cookie 8380515025910090086 [ 3154.326236] LustreError: 217806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3154.327146] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3160.524948] Lustre: server umount lustre-MDT0001 complete [ 3170.286855] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:26:46 (1786544806) [ 3172.889574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3173.001114] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3173.013204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3174.680792] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3178.167608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3178.209591] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3178.222486] Lustre: 226430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3180.051003] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3183.168570] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3185.427450] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3188.114825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3188.202780] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3190.738368] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3193.569908] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3198.183380] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3198.232618] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3200.461445] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3203.328880] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3203.872472] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3205.612902] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3205.721445] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3207.536685] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3207.644163] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3209.926773] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9574884db000.ost_server_uuid 50 [ 3210.624758] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9574884db000.ost_server_uuid in FULL state after 0 sec [ 3211.054603] Lustre: Permanently deactivating lustre-OST0001 [ 3211.059974] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3211.849771] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3211.884027] Lustre: Permanently reactivating lustre-OST0001 [ 3219.139457] Lustre: server umount lustre-OST0001 complete [ 3222.496954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3222.502703] LustreError: Skipped 3 previous similar messages [ 3223.527406] LustreError: 228275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3223.540026] LustreError: 228275:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 28 previous similar messages [ 3235.295102] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3235.356229] Lustre: server umount lustre-OST0000 complete [ 3238.879225] LustreError: 230996:0:(osp_dev.c:699:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 3251.167115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3251.245359] Lustre: server umount lustre-MDT0000 complete [ 3252.845451] LustreError: 226421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544889 with bad export cookie 8380515025910091360 [ 3252.849819] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3252.851590] LustreError: 226421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3252.985090] Lustre: server umount lustre-MDT0001 complete [ 3262.129997] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:28:18 (1786544898) [ 3265.216586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3265.330655] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3265.341468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3265.344925] Lustre: Skipped 5 previous similar messages [ 3266.854499] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3269.927577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3269.983857] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3270.000041] Lustre: 233117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3271.815683] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3275.005110] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3277.002806] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3279.760054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3279.905950] Lustre: Permanently deactivating lustre-OST0000 [ 3279.915794] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3282.568821] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3283.061549] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3283.063460] Lustre: Skipped 1 previous similar message [ 3285.017915] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3287.296716] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3289.282543] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3290.267638] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3291.645427] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3292.191994] Lustre: Permanently reactivating lustre-OST0000 [ 3296.775665] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3296.780377] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3296.786536] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3296.790314] Lustre: Skipped 1 previous similar message [ 3296.810504] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3296.819976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3308.488057] Lustre: server umount lustre-OST0001 complete [ 3324.383205] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3324.470585] Lustre: server umount lustre-OST0000 complete [ 3331.553162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3331.557410] Lustre: Skipped 38 previous similar messages [ 3332.604765] Lustre: server umount lustre-MDT0000 complete [ 3334.341794] LustreError: 233108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786544971 with bad export cookie 8380515025910093082 [ 3334.347468] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3334.347771] LustreError: 233108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3334.485418] Lustre: server umount lustre-MDT0001 complete [ 3338.391200] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 3339.757158] Key type lgssc unregistered [ 3339.915655] LNet: 237975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3339.920644] LNetError: 237975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3339.930563] LNet: Removed LNI 192.168.203.159@tcp [ 3340.444607] Key type .llcrypt unregistered [ 3340.447123] Key type ._llcrypt unregistered [ 3351.497991] Key type ._llcrypt registered [ 3351.499706] Key type .llcrypt registered [ 3351.541512] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3351.900740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3351.951086] alg: No test for adler32 (adler32-zlib) [ 3352.862314] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 3352.970217] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 3354.559231] Key type lgssc registered [ 3355.156626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3360.343962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3360.358078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3361.607410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3363.505437] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3367.843452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3367.906223] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3367.922676] Lustre: 239407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3367.929487] Lustre: Permanently deactivating lustre-MDT0001 [ 3367.937546] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3368.101285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3368.207301] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3368.210126] LustreError: 239449:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3368.214072] LustreError: 239449:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3368.217762] LustreError: 239449:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 3370.011826] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3373.480869] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3376.000897] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3379.373429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3379.519992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3382.306302] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3385.707091] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3386.661063] LustreError: 241259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3386.668516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3386.673534] LustreError: 241259:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3389.115337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3389.229951] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3391.612895] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3393.334662] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3393.336283] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3393.344514] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3395.191478] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3396.875476] Lustre: Permanently reactivating lustre-MDT0001 [ 3396.887949] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3396.892504] Lustre: Skipped 2 previous similar messages [ 3403.628217] Lustre: lustre-MDT0001: Client 9d5bc601-5727-4ef1-9ba5-b08fa9fa407f (at 192.168.203.59@tcp) reconnecting [ 3406.872599] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3407.603738] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3409.839428] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3409.931883] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3411.669488] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3411.739337] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3412.566643] Lustre: Permanently deactivating lustre-MDT0001 [ 3412.571985] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3412.574077] Lustre: Skipped 2 previous similar messages [ 3422.739476] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3424.735936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3424.741650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3424.751318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3427.235288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3427.238433] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.252281] Lustre: Skipped 2 previous similar messages [ 3430.705097] Lustre: server umount lustre-MDT0000 complete [ 3432.119506] LustreError: 239398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786545068 with bad export cookie 10233765192102521728 [ 3432.123815] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3432.124265] LustreError: 239398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3432.253323] Lustre: server umount lustre-MDT0001 complete [ 3439.544706] Lustre: server umount lustre-OST0000 complete [ 3443.649881] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:31:20 (1786545080) [ 3448.530620] Lustre: server umount lustre-OST0001 complete [ 3451.288249] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 3454.941336] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3468.924358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3469.056766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3469.060894] Lustre: Skipped 2 previous similar messages [ 3469.070006] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3469.111407] Lustre: lustre-MDT0000: new disk, initializing [ 3469.132270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3469.138755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3470.784857] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3476.090132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3476.144713] Lustre: 247689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3476.175003] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3476.177949] Lustre: Skipped 1 previous similar message [ 3476.225676] Lustre: lustre-MDT0001: new disk, initializing [ 3476.279934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3476.284423] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3478.122639] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3482.547767] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3484.651632] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3487.666219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3487.840809] Lustre: lustre-OST0000: new disk, initializing [ 3487.844510] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3487.850086] Lustre: 249660:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3487.899992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3487.904737] Lustre: Skipped 1 previous similar message [ 3490.461683] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3494.813386] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3497.471605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3497.477490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3497.529292] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3500.985340] LustreError: 247697:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3505.900364] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3505.962580] Lustre: lustre-OST0001: new disk, initializing [ 3505.964273] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3505.966608] Lustre: 250775:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3505.995880] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3508.305731] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3511.079066] LustreError: 247697:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3511.087812] LustreError: 247696:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3512.717398] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3513.366177] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3515.393852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3515.397723] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3515.433370] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3521.175109] LustreError: 247696:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3522.920869] Lustre: lustre-OST0001: Not available for connect from 192.168.203.59@tcp (stopping) [ 3522.923760] Lustre: Skipped 3 previous similar messages [ 3525.610186] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3525.611099] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3525.614860] LustreError: Skipped 1 previous similar message [ 3525.624238] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3525.629216] Lustre: Skipped 1 previous similar message [ 3530.724183] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3530.727483] Lustre: Skipped 2 previous similar messages [ 3536.351147] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3536.401852] Lustre: server umount lustre-OST0001 complete [ 3540.966451] LustreError: 251632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3540.981096] LustreError: 251632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3546.083243] LustreError: 249644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3546.091612] LustreError: 249644:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3551.202764] LustreError: 249646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3551.202957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3551.205116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3551.205124] Lustre: Skipped 3 previous similar messages [ 3551.209794] LustreError: 249646:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3551.218193] Lustre: Skipped 2 previous similar messages [ 3554.895201] Lustre: server umount lustre-OST0000 complete [ 3556.320738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3556.324349] Lustre: Skipped 1 previous similar message [ 3562.498486] Lustre: server umount lustre-MDT0000 complete [ 3564.369171] LustreError: 247683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786545201 with bad export cookie 10233765192102524507 [ 3564.376398] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3564.378096] LustreError: 247683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3570.554854] Lustre: server umount lustre-MDT0001 complete [ 3575.434588] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 3577.037775] Key type lgssc unregistered [ 3577.221520] LNet: 252810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3577.226851] LNetError: 252810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3577.239579] LNet: Removed LNI 192.168.203.159@tcp [ 3577.757592] Key type .llcrypt unregistered [ 3577.759482] Key type ._llcrypt unregistered [ 3596.088206] Key type ._llcrypt registered [ 3596.090279] Key type .llcrypt registered [ 3596.148994] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3596.623508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3596.676094] alg: No test for adler32 (adler32-zlib) [ 3597.563460] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 3597.702749] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 3599.319168] Key type lgssc registered [ 3600.033698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3604.289443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3604.303762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3605.431255] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3605.444270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3605.525049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3607.270543] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3610.991596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3611.037189] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3611.056355] Lustre: 254991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3611.193712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3612.987138] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3616.272841] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3618.758475] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3621.575402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3621.675297] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3621.762418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3624.046816] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3627.085729] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3631.653651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3672.550280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3672.550753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3672.553312] LustreError: Skipped 1 previous similar message [ 3672.557974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3672.560083] Lustre: Skipped 1 previous similar message [ 3677.668620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3677.670939] Lustre: Skipped 1 previous similar message [ 3677.917482] Lustre: server umount lustre-OST0000 complete [ 3679.991460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3686.129895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3686.334201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3688.879391] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3691.866868] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3702.698625] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3702.711234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3702.711326] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3702.724656] Lustre: Skipped 1 previous similar message [ 3728.355629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3728.357379] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3728.360052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3728.360684] LustreError: Skipped 1 previous similar message [ 3728.369534] Lustre: Skipped 2 previous similar messages [ 3731.880151] Lustre: server umount lustre-OST0000 complete [ 3733.474775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3733.475519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3733.482486] Lustre: Skipped 1 previous similar message [ 3733.486427] Lustre: Skipped 1 previous similar message [ 3738.593591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3738.597964] Lustre: Skipped 1 previous similar message [ 3739.641929] Lustre: server umount lustre-MDT0000 complete [ 3741.282291] LustreError: 254984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786545377 with bad export cookie 3380190843046291572 [ 3741.285345] LustreError: 254984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3741.285890] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3741.398618] Lustre: server umount lustre-MDT0001 complete [ 3745.075224] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 3746.018674] Key type lgssc unregistered [ 3746.177529] LNet: 260410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3746.180245] LNetError: 260410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3746.191446] LNet: Removed LNI 192.168.203.159@tcp [ 3746.563243] Key type .llcrypt unregistered [ 3746.564662] Key type ._llcrypt unregistered [ 3756.232912] Key type ._llcrypt registered [ 3756.233891] Key type .llcrypt registered [ 3756.268185] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3756.582823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3756.617403] alg: No test for adler32 (adler32-zlib) [ 3757.462343] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 3757.542572] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 3759.119109] Key type lgssc registered [ 3759.414436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3761.608582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3761.614246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3762.765976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3763.944566] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3766.521041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3766.638708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3767.896955] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3770.974585] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3773.441160] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3776.695571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3776.888791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3779.038977] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3781.763393] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3788.139490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3803.619811] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3803.623057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3803.626455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3803.632918] Lustre: Skipped 1 previous similar message [ 3807.465655] Lustre: server umount lustre-OST0000 complete [ 3812.832740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3812.838046] LustreError: Skipped 1 previous similar message [ 3812.841083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3812.850173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3812.854401] Lustre: Skipped 1 previous similar message [ 3813.857347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3813.858477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3815.537834] Lustre: server umount lustre-MDT0000 complete [ 3816.982580] LustreError: 263644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786545453 with bad export cookie 8010636760859663080 [ 3816.987545] LustreError: 263644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3816.987916] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3817.124788] Lustre: server umount lustre-MDT0001 complete [ 3821.224705] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 3822.476719] Key type lgssc unregistered [ 3822.608750] LNet: 266131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3822.613488] LNetError: 266131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3822.623641] LNet: Removed LNI 192.168.203.159@tcp [ 3823.100135] Key type .llcrypt unregistered [ 3823.102359] Key type ._llcrypt unregistered [ 3830.715603] Key type ._llcrypt registered [ 3830.716683] Key type .llcrypt registered [ 3830.771687] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3831.066829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3831.080673] alg: No test for adler32 (adler32-zlib) [ 3831.931161] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 3832.013889] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 3833.591197] Key type lgssc registered [ 3833.995698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3836.899219] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3836.906061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3838.058158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3839.570176] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3843.257400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3843.393621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3845.013318] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3847.766978] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3849.948584] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3853.082270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3853.220405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3855.623612] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3858.503676] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3862.446618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3867.618474] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3867.618710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3867.619379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3867.621860] Lustre: Skipped 1 previous similar message [ 3869.131979] Lustre: server umount lustre-OST0000 complete [ 3872.601838] Lustre: server umount lustre-MDT0000 complete [ 3872.737467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3872.738335] LustreError: 268577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3872.741266] Lustre: Skipped 1 previous similar message [ 3872.745337] LustreError: 268577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3873.799209] LustreError: 267312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786545510 with bad export cookie 12892659978501985982 [ 3873.802230] LustreError: 267312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3873.802565] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3873.917270] Lustre: server umount lustre-MDT0001 complete [ 3877.131284] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 3878.044096] Key type lgssc unregistered [ 3878.159381] LNet: 271006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3878.161649] LNetError: 271006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3878.173277] LNet: Removed LNI 192.168.203.159@tcp [ 3878.468073] Key type .llcrypt unregistered [ 3878.469151] Key type ._llcrypt unregistered [ 3886.530059] Key type ._llcrypt registered [ 3886.531132] Key type .llcrypt registered [ 3886.565324] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3886.909333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3886.938425] alg: No test for adler32 (adler32-zlib) [ 3887.779848] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 3887.866988] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 3889.439152] Key type lgssc registered [ 3889.741614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3891.984067] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3891.990362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3893.129350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3894.273707] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3896.983477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3897.164970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3898.787723] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3901.376539] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3903.284101] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3906.204980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3906.377845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3908.484693] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3911.366199] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3918.702763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3933.152256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3933.155705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3933.160808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3934.176639] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3934.177280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3934.178816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3936.081130] Lustre: server umount lustre-OST0000 complete [ 3938.271942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3938.274235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3938.280456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3938.282224] Lustre: Skipped 1 previous similar message [ 3943.476354] Lustre: server umount lustre-MDT0000 complete [ 3944.417511] LustreError: 272398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3944.430082] LustreError: 272398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3944.883185] LustreError: 272380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786545581 with bad export cookie 14214036410024437494 [ 3944.887570] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3944.889028] LustreError: 272380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3945.008442] Lustre: server umount lustre-MDT0001 complete [ 3948.690604] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 3949.888064] Key type lgssc unregistered [ 3950.030663] LNet: 276725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3950.035882] LNetError: 276725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3950.046518] LNet: Removed LNI 192.168.203.159@tcp [ 3950.510520] Key type .llcrypt unregistered [ 3950.513068] Key type ._llcrypt unregistered [ 3958.205284] Key type ._llcrypt registered [ 3958.206540] Key type .llcrypt registered [ 3958.245947] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3958.584808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3958.608307] alg: No test for adler32 (adler32-zlib) [ 3959.463928] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 3959.549814] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 3961.135223] Key type lgssc registered [ 3961.627590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3964.738674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3964.747608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3965.936587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3967.334054] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3970.621847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3970.754906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3972.076514] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3974.520629] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3976.304165] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3978.529045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3978.645237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3980.538602] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3982.817567] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3990.950270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4001.252363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4001.255087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4001.256524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4001.262631] Lustre: Skipped 1 previous similar message [ 4002.907582] Lustre: server umount lustre-OST0000 complete [ 4006.368856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4006.373679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4006.376380] Lustre: Skipped 2 previous similar messages [ 4010.787404] Lustre: server umount lustre-MDT0000 complete [ 4011.487865] LustreError: 277950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4011.492387] LustreError: 277950:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4012.000077] LustreError: 277904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786545648 with bad export cookie 17103518288812959310 [ 4012.004708] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4012.006200] LustreError: 277904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4012.155173] Lustre: server umount lustre-MDT0001 complete [ 4016.069300] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4017.173366] Key type lgssc unregistered [ 4017.310455] LNet: 281657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4017.315472] LNetError: 281657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4017.328510] LNet: Removed LNI 192.168.203.159@tcp [ 4017.681996] Key type .llcrypt unregistered [ 4017.683638] Key type ._llcrypt unregistered [ 4021.599363] Key type ._llcrypt registered [ 4021.600438] Key type .llcrypt registered [ 4021.638678] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4028.154662] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 4034.553428] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4034.876192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4034.923609] alg: No test for adler32 (adler32-zlib) [ 4035.769525] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 4035.852044] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 4037.423260] Key type lgssc registered [ 4037.794578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4050.547496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4050.558934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4051.656371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4051.671172] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4051.715379] Lustre: lustre-MDT0000: new disk, initializing [ 4051.736677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4051.742935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4053.138923] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4057.925947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4057.978287] Lustre: 285180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4058.016827] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4058.020256] Lustre: Skipped 1 previous similar message [ 4058.064625] Lustre: lustre-MDT0001: new disk, initializing [ 4058.118424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4058.132175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4058.135268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4059.460553] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4063.431929] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4065.212736] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4067.530407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4067.646720] Lustre: lustre-OST0000: new disk, initializing [ 4067.648391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4067.650747] Lustre: 287147:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4067.678591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4069.712760] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4074.372741] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4075.533330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4075.541163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4075.578818] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4076.185764] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4076.260800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4077.586571] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4077.675505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4080.610876] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4080.611724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4080.615813] LustreError: Skipped 1 previous similar message [ 4080.620923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4080.624909] Lustre: Skipped 1 previous similar message [ 4084.531098] Lustre: server umount lustre-OST0000 complete [ 4088.799784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4088.803034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4088.807970] Lustre: Skipped 1 previous similar message [ 4088.810177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4090.848697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4090.849127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4092.131177] Lustre: server umount lustre-MDT0000 complete [ 4093.513161] LustreError: 285172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786545730 with bad export cookie 13668824289001679089 [ 4093.516621] LustreError: 285172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4093.516749] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4093.638171] Lustre: server umount lustre-MDT0001 complete [ 4096.513675] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:42:13 (1786545733) [ 4098.878404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4099.076253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4100.495843] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4103.385777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4103.514899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4104.827172] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4107.091283] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4108.834754] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4111.005223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4112.927606] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4115.270514] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4129.760369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4129.765043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4129.772322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4129.775821] Lustre: Skipped 1 previous similar message [ 4133.857775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4133.857838] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4133.857871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4133.859487] Lustre: Skipped 1 previous similar message [ 4134.987853] Lustre: server umount lustre-OST0000 complete [ 4142.294606] Lustre: server umount lustre-MDT0000 complete [ 4143.412100] LustreError: 291401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786545780 with bad export cookie 13668824289001679957 [ 4143.414147] LustreError: 290294:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4143.414738] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4143.415312] LustreError: 291401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4143.534837] Lustre: server umount lustre-MDT0001 complete [ 4146.753476] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4147.679029] Key type lgssc unregistered [ 4147.806306] LNet: 293115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4147.808416] LNetError: 293115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4147.819291] LNet: Removed LNI 192.168.203.159@tcp [ 4148.099761] Key type .llcrypt unregistered [ 4148.101506] Key type ._llcrypt unregistered [ 4155.931043] Key type ._llcrypt registered [ 4155.933194] Key type .llcrypt registered [ 4155.968497] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 4161.729289] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4162.043690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4162.051477] alg: No test for adler32 (adler32-zlib) [ 4162.896708] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 4162.989367] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 4164.567156] Key type lgssc registered [ 4164.903287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4179.736570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4179.743253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4180.828857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4180.838351] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4180.872691] Lustre: lustre-MDT0000: new disk, initializing [ 4180.892905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4180.898453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4182.292253] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4187.303317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4187.380090] Lustre: 296912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4187.422668] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4187.425861] Lustre: Skipped 1 previous similar message [ 4187.465494] Lustre: lustre-MDT0001: new disk, initializing [ 4187.500751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4187.515060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4187.520124] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4188.930086] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4193.304746] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4195.055092] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4197.598676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4197.700007] Lustre: lustre-OST2710: new disk, initializing [ 4197.702334] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4197.705231] Lustre: 298878:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4197.741490] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4200.110723] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4203.466754] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4204.534902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4204.537711] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4204.567694] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4205.527469] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4205.585066] Lustre: lustre-OST03e8: new disk, initializing [ 4205.588184] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4205.592618] Lustre: 299945:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4205.620348] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4207.378358] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4209.530018] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4210.690975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4210.694064] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4210.725324] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4211.927397] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4211.989265] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4213.327315] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4213.409915] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4214.780222] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4214.850549] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4216.130707] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4216.207566] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4218.335963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4218.338289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4218.342656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4220.900183] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4220.900797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4220.908729] Lustre: Skipped 3 previous similar messages [ 4223.324509] Lustre: server umount lustre-MDT0000 complete [ 4224.489961] LustreError: 297706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786545861 with bad export cookie 9690162884528658185 [ 4224.494369] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4224.494883] LustreError: 297706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4224.605475] Lustre: server umount lustre-MDT0001 complete [ 4236.230814] Lustre: server umount lustre-OST2710 complete [ 4247.641883] Lustre: server umount lustre-OST03e8 complete [ 4253.093549] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 4255.910641] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4268.422252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4268.506200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4268.515250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4268.555184] Lustre: lustre-MDT0000: new disk, initializing [ 4268.576708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4268.581849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4269.888417] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4274.644863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4274.684861] Lustre: 305191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4274.782367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4276.044594] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4279.594177] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4281.413541] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4283.581079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4283.683356] Lustre: lustre-OST0000: new disk, initializing [ 4283.684645] Lustre: Skipped 1 previous similar message [ 4283.685976] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4283.687576] Lustre: Skipped 2 previous similar messages [ 4283.689334] Lustre: 307162:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4283.717558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4283.719865] Lustre: Skipped 1 previous similar message [ 4285.848301] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4287.996023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4287.998749] Lustre: Skipped 1 previous similar message [ 4288.000732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4288.037032] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4289.891468] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4291.453989] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4291.532742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4293.058289] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4293.145926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4298.211163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4298.212054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4298.215761] LustreError: Skipped 1 previous similar message [ 4298.226552] Lustre: Skipped 2 previous similar messages [ 4298.227908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4298.235321] Lustre: Skipped 1 previous similar message [ 4299.868909] Lustre: server umount lustre-OST0000 complete [ 4303.328500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4303.329255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4303.331650] Lustre: Skipped 2 previous similar messages [ 4303.334641] Lustre: Skipped 1 previous similar message [ 4307.374088] Lustre: server umount lustre-MDT0000 complete [ 4308.571297] LustreError: 307161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786545945 with bad export cookie 9690162884528659865 [ 4308.574059] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4308.575690] LustreError: 307161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4308.689145] Lustre: server umount lustre-MDT0001 complete [ 4311.334201] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 10:45:48 (1786545948) [ 4311.818919] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4312.352516] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 10:45:49 (1786545949) [ 4313.568762] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4325.207895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4325.287895] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4325.294219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4325.346028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4326.820985] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4329.964679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4329.999417] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4330.011340] Lustre: 311013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4331.753077] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4348.261216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4348.378609] Lustre: Denying initial registration attempt from nid 192.168.203.159@tcp, specified as failover [ 4348.383699] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4348.388920] LustreError: 312466:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 4348.392715] LustreError: 312466:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4348.395497] LustreError: 312466:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4348.430815] Lustre: server umount lustre-OST0000 complete [ 4348.433329] LustreError: 312466:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4351.969085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4351.970229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4351.978387] Lustre: Skipped 1 previous similar message [ 4357.293992] Lustre: server umount lustre-MDT0000 complete [ 4358.982272] LustreError: 311004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786545995 with bad export cookie 9690162884528660747 [ 4358.990024] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4358.990117] LustreError: 311004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4359.155669] Lustre: server umount lustre-MDT0001 complete [ 4364.118692] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4365.454794] Key type lgssc unregistered [ 4365.596752] LNet: 313589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4365.605370] LNetError: 313589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4365.618624] LNet: Removed LNI 192.168.203.159@tcp [ 4366.090741] Key type .llcrypt unregistered [ 4366.093540] Key type ._llcrypt unregistered [ 4369.873500] Key type ._llcrypt registered [ 4369.874665] Key type .llcrypt registered [ 4369.915762] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4370.215678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4370.254254] alg: No test for adler32 (adler32-zlib) [ 4371.104112] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 4371.185120] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 4372.759222] Key type lgssc registered [ 4373.178827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4399.625483] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4402.986067] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4402.997149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4404.098941] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4404.106878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4404.183827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4405.865386] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4409.620989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4409.658311] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4409.670084] Lustre: 316327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4409.765296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4411.241089] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4414.335076] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4416.647758] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4432.483835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4432.562972] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4432.625162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4434.837118] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4437.611653] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4438.581080] Lustre: server umount lustre-OST0000 complete [ 4440.544488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4440.549392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4440.557844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4441.083087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4441.083980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4446.176934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4446.181510] Lustre: Skipped 2 previous similar messages [ 4451.297302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4451.301173] Lustre: Skipped 1 previous similar message [ 4454.367074] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4454.437670] Lustre: server umount lustre-MDT0000 complete [ 4455.814793] LustreError: 316320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786546092 with bad export cookie 12244047893435847298 [ 4455.820114] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4455.820726] LustreError: 316320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4455.985139] Lustre: server umount lustre-MDT0001 complete [ 4460.275481] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4461.326654] Key type lgssc unregistered [ 4461.441361] LNet: 319893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4461.443601] LNetError: 319893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4461.452291] LNet: Removed LNI 192.168.203.159@tcp [ 4461.730250] Key type .llcrypt unregistered [ 4461.731261] Key type ._llcrypt unregistered [ 4471.028928] Key type ._llcrypt registered [ 4471.030367] Key type .llcrypt registered [ 4471.063262] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4471.455438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4471.465799] alg: No test for adler32 (adler32-zlib) [ 4472.317337] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 4472.410612] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 4473.991183] Key type lgssc registered [ 4474.344669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4477.572076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4477.580624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4478.744496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4480.470319] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4484.438755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4484.622339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4486.587250] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4490.073464] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4492.710107] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4498.148612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4498.332329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4500.850508] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4504.057168] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4514.785695] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4514.787144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4514.790718] Lustre: Skipped 2 previous similar messages [ 4514.792413] Lustre: Skipped 1 previous similar message [ 4515.687677] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.59@tcp (stopping) [ 4519.026891] Lustre: server umount lustre-MDT0000 complete [ 4519.906292] LustreError: 323726:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4519.910814] LustreError: 323726:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4520.800093] LustreError: 321281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4520.819084] LustreError: 321267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786546157 with bad export cookie 15732522734260884955 [ 4520.825899] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4520.826357] LustreError: 321267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4525.024215] LustreError: 321282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4525.024619] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4525.039771] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4530.144599] LustreError: 321311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4530.144932] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4530.155530] LustreError: 321311:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4530.167461] Lustre: Skipped 1 previous similar message [ 4535.263194] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4535.352164] Lustre: server umount lustre-MDT0001 complete [ 4538.249654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4541.682678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4545.503856] LustreError: 320598:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000806801d6 x1873329372816512/t0(0) o250->MGC192.168.203.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4545.666675] LustreError: 323114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4545.674463] LustreError: 323114:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4545.700546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4545.740438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4547.472756] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4551.473638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4551.663814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4553.011443] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4556.235911] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4560.870437] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4563.047216] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 4564.898383] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4565.576460] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4571.107827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4571.111076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4571.115834] Lustre: Skipped 1 previous similar message [ 4571.119696] Lustre: Skipped 2 previous similar messages [ 4573.397990] Lustre: server umount lustre-OST0000 complete [ 4576.225962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4581.111349] Lustre: server umount lustre-MDT0000 complete [ 4581.345147] LustreError: 324879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4581.356040] LustreError: 324879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4582.519355] LustreError: 323127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786546219 with bad export cookie 15732522734260892074 [ 4582.524488] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4582.526422] LustreError: 323127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4582.670894] Lustre: server umount lustre-MDT0001 complete [ 4586.839930] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4587.748378] Key type lgssc unregistered [ 4587.870362] LNet: 327548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4587.875644] LNetError: 327548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4587.885480] LNet: Removed LNI 192.168.203.159@tcp [ 4588.266792] Key type .llcrypt unregistered [ 4588.268680] Key type ._llcrypt unregistered [ 4598.483189] Key type ._llcrypt registered [ 4598.484221] Key type .llcrypt registered [ 4598.521342] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4598.906773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4598.917284] alg: No test for adler32 (adler32-zlib) [ 4599.757340] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 4599.841769] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 4601.415279] Key type lgssc registered [ 4601.851064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4604.890220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4604.906544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4606.068769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4607.410073] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4610.518419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4610.637978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4612.120116] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4615.237199] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4617.154165] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4620.320884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4620.321462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4620.326062] Lustre: Skipped 1 previous similar message [ 4624.497744] Lustre: server umount lustre-MDT0000 complete [ 4625.377084] LustreError: 328941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4625.388017] LustreError: 328941:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4625.936322] LustreError: 328923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786546262 with bad export cookie 11355808277005086922 [ 4625.939509] LustreError: 328923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4625.939715] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4626.067435] Lustre: server umount lustre-MDT0001 complete [ 4631.980896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4632.087856] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4632.098852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4632.174805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4633.854547] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4637.675691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4637.717564] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4637.729422] Lustre: 331526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4637.830596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4639.299661] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4641.749198] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4644.107792] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4649.444828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4649.563642] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4649.571337] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4649.578692] LustreError: 333409:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 4649.584600] LustreError: 333409:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4649.589985] LustreError: 333409:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4649.629671] Lustre: server umount lustre-OST0000 complete [ 4649.631831] LustreError: 333409:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4655.329995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4655.404516] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4655.471317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4657.755853] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4660.723357] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4663.326864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4663.746324] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4663.780619] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4663.815481] Lustre: lustre-OST0001: new disk, initializing [ 4663.817939] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4663.822257] Lustre: 334947:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4663.853083] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4665.663703] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4667.848168] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4668.775547] Lustre: server umount lustre-OST0001 complete [ 4668.920283] LustreError: 333875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4673.504361] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4673.504901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4673.507918] LustreError: Skipped 1 previous similar message [ 4673.515955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4673.522548] Lustre: Skipped 2 previous similar messages [ 4674.019710] LustreError: 333874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4674.024068] LustreError: 333874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4679.138939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4679.139913] LustreError: 333874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4679.142143] Lustre: Skipped 3 previous similar messages [ 4679.147406] LustreError: 333874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4684.255328] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4684.259696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4684.260086] LustreError: 333875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4684.265966] Lustre: Skipped 1 previous similar message [ 4684.306394] Lustre: server umount lustre-OST0000 complete [ 4689.376818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4689.376923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4689.380292] Lustre: Skipped 1 previous similar message [ 4689.385157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4689.386779] Lustre: Skipped 1 previous similar message [ 4691.736668] Lustre: server umount lustre-MDT0000 complete [ 4692.886062] LustreError: 331519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786546329 with bad export cookie 11355808277005087461 [ 4692.889147] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4692.889372] LustreError: 331519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4693.001443] Lustre: server umount lustre-MDT0001 complete [ 4696.222328] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4697.144671] Key type lgssc unregistered [ 4697.268362] LNet: 336907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4697.272219] LNetError: 336907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4697.281328] LNet: Removed LNI 192.168.203.159@tcp [ 4697.560108] Key type .llcrypt unregistered [ 4697.562070] Key type ._llcrypt unregistered [ 4725.989663] Key type ._llcrypt registered [ 4725.990670] Key type .llcrypt registered [ 4726.021833] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 4731.695718] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4732.100879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4732.106198] alg: No test for adler32 (adler32-zlib) [ 4732.950355] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 4733.030658] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 4734.607112] Key type lgssc registered [ 4734.905855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4744.996939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4745.003492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4746.076854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4746.085656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4746.119565] Lustre: lustre-MDT0000: new disk, initializing [ 4746.139100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4746.145141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4747.374988] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4751.235239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4751.274649] Lustre: 341811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4751.297628] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4751.299334] Lustre: Skipped 1 previous similar message [ 4751.335032] Lustre: lustre-MDT0001: new disk, initializing [ 4751.363606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4751.373817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4751.376707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4752.591085] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4756.014518] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4757.657476] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4759.821052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4759.913949] Lustre: lustre-OST0000: new disk, initializing [ 4759.915550] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4759.917792] Lustre: 343775:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4759.942824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4761.960493] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4762.613945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4762.616666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4762.645696] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4765.396117] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4767.222363] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4767.322554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4769.043738] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4769.113670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4772.837126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4772.838338] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4772.842547] LustreError: Skipped 1 previous similar message [ 4772.852434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4772.858890] Lustre: Skipped 1 previous similar message [ 4776.079757] Lustre: server umount lustre-OST0000 complete [ 4777.952676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4777.953365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4777.956787] Lustre: Skipped 2 previous similar messages [ 4783.072815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4783.075900] Lustre: Skipped 2 previous similar messages [ 4783.770429] Lustre: server umount lustre-MDT0000 complete [ 4785.216876] LustreError: 341801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786546421 with bad export cookie 8586686674502980521 [ 4785.222056] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4785.228978] LustreError: 341801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4785.364172] Lustre: server umount lustre-MDT0001 complete [ 4788.788818] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 10:53:45 (1786546425) [ 4791.657002] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 10:53:48 (1786546428) [ 4794.658618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4794.826436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4796.430520] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4799.968973] LustreError: 346528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4799.984599] LustreError: 346528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4800.576212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4800.741276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4802.522907] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4805.819326] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4808.161298] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4810.746335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4810.883166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4812.806309] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4815.305802] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4824.159673] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4824.802648] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4825.381086] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4826.055709] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4826.888605] Lustre: Failing over lustre-MDT0000 [ 4827.165582] Lustre: server umount lustre-MDT0000 complete [ 4830.690741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4830.692943] LustreError: 346522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4830.701771] Lustre: Skipped 2 previous similar messages [ 4830.711479] LustreError: 346522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4835.808205] LustreError: 346546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4835.814845] LustreError: 346546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4840.588230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4840.654339] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4840.771417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4840.793571] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4841.312490] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4842.297621] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4846.064563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4846.075846] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4846.100937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4847.915429] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4848.417777] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4850.295484] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4850.998561] Lustre: Failing over lustre-MDT0000 [ 4851.170498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4851.171508] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4851.173758] Lustre: Skipped 1 previous similar message [ 4851.176193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4851.180649] Lustre: Skipped 2 previous similar messages [ 4851.268398] Lustre: server umount lustre-MDT0000 complete [ 4851.556842] LustreError: 346523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4851.564300] LustreError: 346523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4856.289198] LustreError: 346528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4856.300261] LustreError: 346528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4864.774227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.828339] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4864.971348] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4866.459594] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4866.914178] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4870.119879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4870.122996] Lustre: Skipped 2 previous similar messages [ 4870.142534] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4870.174155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4872.278291] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4872.928141] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4880.354338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4880.354453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4880.358533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4880.362962] Lustre: Skipped 3 previous similar messages [ 4881.462859] Lustre: server umount lustre-MDT0000 complete [ 4882.783099] LustreError: 346508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786546519 with bad export cookie 8586686674502983202 [ 4882.785391] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4882.786119] LustreError: 346508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4882.894487] Lustre: server umount lustre-MDT0001 complete [ 4894.387388] Lustre: server umount lustre-OST0000 complete [ 4900.744195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4900.930551] LustreError: 352855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4900.936294] LustreError: 352855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4900.956642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4900.958916] Lustre: Skipped 1 previous similar message [ 4902.757856] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4906.631460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4908.527681] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4911.096571] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4913.510516] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4916.303994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4918.153603] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4920.365957] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4925.670469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4927.967865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4927.970382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4927.975046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4927.977404] Lustre: Skipped 2 previous similar messages [ 4932.903265] Lustre: server umount lustre-OST0000 complete [ 4940.378801] Lustre: server umount lustre-MDT0000 complete [ 4941.279877] LustreError: 352855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4941.284198] LustreError: 352855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4941.658410] LustreError: 352834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786546578 with bad export cookie 8586686674502983944 [ 4941.662142] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4941.663545] LustreError: 352834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4941.772708] Lustre: server umount lustre-MDT0001 complete [ 4945.590711] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4946.496630] Key type lgssc unregistered [ 4946.617641] LNet: 356391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4946.621667] LNetError: 356391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4946.634376] LNet: Removed LNI 192.168.203.159@tcp [ 4947.002206] Key type .llcrypt unregistered [ 4947.004598] Key type ._llcrypt unregistered [ 4957.516857] Key type ._llcrypt registered [ 4957.517945] Key type .llcrypt registered [ 4957.555752] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4957.872042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4957.929470] alg: No test for adler32 (adler32-zlib) [ 4958.788614] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 4958.887912] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 4960.471189] Key type lgssc registered [ 4960.927467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4964.352756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4964.358193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4965.519140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4966.936756] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4969.495881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4969.608847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4970.965686] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4973.342909] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4975.062752] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4977.122373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4977.224977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4978.961219] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4981.155595] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4990.565259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 4995.551860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4995.553922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4995.557357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4996.064756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4996.066188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5001.188676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5001.193350] Lustre: Skipped 4 previous similar messages [ 5001.632714] Lustre: server umount lustre-MDT0000 complete [ 5002.838237] LustreError: 357864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786546639 with bad export cookie 17590704206355657890 [ 5002.841459] LustreError: 357864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5002.842265] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5002.952500] Lustre: server umount lustre-MDT0001 complete [ 5014.697808] Lustre: server umount lustre-OST0000 complete [ 5019.513047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5019.652041] LustreError: 361613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5019.657405] LustreError: 361613:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5019.673797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5020.995355] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5024.056740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5024.195579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5025.794472] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5028.354905] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5030.489266] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5033.280077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5033.411979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5035.565080] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5037.995541] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5044.711511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5060.067059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5060.068980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5060.069286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5060.073033] Lustre: Skipped 2 previous similar messages [ 5061.742944] Lustre: server umount lustre-MDT0000 complete [ 5063.292704] LustreError: 361593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786546699 with bad export cookie 17590704206355658989 [ 5063.297277] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5063.297512] LustreError: 361593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5063.429988] Lustre: server umount lustre-MDT0001 complete [ 5074.999614] Lustre: server umount lustre-OST0000 complete [ 5081.032970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5081.187935] LustreError: 365545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5081.212841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5082.775042] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5086.483484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5088.407258] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5091.051270] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5093.597186] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5096.420156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5098.592931] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5101.509488] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5105.766049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5108.191902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5108.195151] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5108.200450] Lustre: Skipped 1 previous similar message [ 5108.202100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5108.203899] Lustre: Skipped 2 previous similar messages [ 5111.268218] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5114.055726] Lustre: server umount lustre-OST0000 complete [ 5116.384624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5116.387848] Lustre: Skipped 1 previous similar message [ 5116.389552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5116.391151] Lustre: Skipped 3 previous similar messages [ 5121.297344] Lustre: server umount lustre-MDT0000 complete [ 5121.505059] LustreError: 365564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5121.512348] LustreError: 365564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5122.397963] LustreError: 365527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786546759 with bad export cookie 17590704206355659997 [ 5122.400400] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5122.400977] LustreError: 365527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5122.511802] Lustre: server umount lustre-MDT0001 complete [ 5125.664481] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 5126.597228] Key type lgssc unregistered [ 5126.706379] LNet: 369075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5126.708701] LNetError: 369075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5126.720297] LNet: Removed LNI 192.168.203.159@tcp [ 5126.990888] Key type .llcrypt unregistered [ 5126.991867] Key type ._llcrypt unregistered [ 5135.575493] Key type ._llcrypt registered [ 5135.576485] Key type .llcrypt registered [ 5135.607105] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 5135.922128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5135.948061] alg: No test for adler32 (adler32-zlib) [ 5136.793600] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 5136.885848] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 5138.471116] Key type lgssc registered [ 5138.811240] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5141.192934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5141.198275] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5141.200588] LustreError: 370492:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5141.207950] LustreError: 370492:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5141.211167] LustreError: 370492:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5141.214910] LustreError: 370492:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5141.218389] LustreError: 370492:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5143.504041] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5143.506365] LustreError: 370889:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5143.513698] LustreError: 370889:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5143.515824] LustreError: 370889:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5143.517904] LustreError: 370889:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5143.520329] LustreError: 370889:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5149.227737] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 5150.123211] Key type lgssc unregistered [ 5150.233310] LNet: 371892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5150.235577] LNetError: 371892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5150.246241] LNet: Removed LNI 192.168.203.159@tcp [ 5150.512311] Key type .llcrypt unregistered [ 5150.513364] Key type ._llcrypt unregistered [ 5156.684985] Key type ._llcrypt registered [ 5156.685945] Key type .llcrypt registered [ 5156.725370] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 5162.488404] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 5162.785366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5162.814248] alg: No test for adler32 (adler32-zlib) [ 5163.685075] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 5163.781736] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 5165.367148] Key type lgssc registered [ 5165.723593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5178.731212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5178.740578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5179.841991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5179.851408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5179.887295] Lustre: lustre-MDT0000: new disk, initializing [ 5179.910040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5179.916607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5181.842592] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5187.466464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5187.538582] Lustre: 375020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5187.578996] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5187.582572] Lustre: Skipped 1 previous similar message [ 5187.632695] Lustre: lustre-MDT0001: new disk, initializing [ 5187.668680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5187.681985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5187.684899] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5189.161860] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5193.935562] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5196.463893] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5199.115538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5199.231493] Lustre: lustre-OST0000: new disk, initializing [ 5199.233253] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5199.235786] Lustre: 376988:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5199.263357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5201.668180] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5202.701950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5202.709656] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5202.744822] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5205.582194] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5206.893778] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5206.971277] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5208.478248] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5208.553375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5213.156984] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5213.157740] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5213.165558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5213.174334] Lustre: Skipped 1 previous similar message [ 5215.313059] Lustre: server umount lustre-OST0000 complete [ 5218.274577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5218.275174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5218.279462] LustreError: Skipped 1 previous similar message [ 5218.286716] Lustre: Skipped 1 previous similar message [ 5218.293161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5218.296698] Lustre: Skipped 1 previous similar message [ 5222.907410] Lustre: server umount lustre-MDT0000 complete [ 5224.243570] LustreError: 375014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786546860 with bad export cookie 10351844535413314989 [ 5224.248819] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5224.249050] LustreError: 375014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5224.373926] Lustre: server umount lustre-MDT0001 complete [ 5227.397873] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5227.935916] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:01:04 (1786546864) [ 5230.381898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5230.531187] LustreError: 379453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5230.539912] LustreError: 379453:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5230.554079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5231.852348] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5235.030047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5235.154776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5236.641086] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5239.477464] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5241.672381] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5244.269601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5244.384193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5246.556750] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5249.487518] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5252.071201] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5252.125638] Lustre: lustre-OST0001: new disk, initializing [ 5252.127195] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5252.129380] Lustre: 382289:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5254.386748] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5254.615979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5254.622891] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5254.671887] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5258.911811] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5261.799267] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5261.802961] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5261.807134] Lustre: Skipped 2 previous similar messages [ 5261.814099] Lustre: Skipped 1 previous similar message [ 5265.263213] Lustre: lustre-OST0001: Not available for connect from 192.168.203.59@tcp (stopping) [ 5267.131982] Lustre: server umount lustre-OST0001 complete [ 5272.034801] LustreError: 381274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5272.038750] LustreError: 381274:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5277.161511] LustreError: 381932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5277.171192] LustreError: 381932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5280.224059] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5280.227501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5280.233075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5280.235832] Lustre: Skipped 2 previous similar messages [ 5282.277497] LustreError: 381273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5282.279514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5282.287976] LustreError: 381273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5285.257893] Lustre: server umount lustre-OST0000 complete [ 5286.367601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5286.369743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5286.372857] Lustre: Skipped 1 previous similar message [ 5292.512809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5292.516255] Lustre: Skipped 6 previous similar messages [ 5292.593677] Lustre: server umount lustre-MDT0000 complete [ 5294.045448] LustreError: 379434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786546930 with bad export cookie 10351844535413315871 [ 5294.048628] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5294.049489] LustreError: 379434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5308.383198] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5308.488236] Lustre: server umount lustre-MDT0001 complete [ 5312.422134] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 5313.349530] Key type lgssc unregistered [ 5313.470368] LNet: 384278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5313.474318] LNetError: 384278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5313.484318] LNet: Removed LNI 192.168.203.159@tcp [ 5313.800962] Key type .llcrypt unregistered [ 5313.802028] Key type ._llcrypt unregistered [ 5335.524239] Key type ._llcrypt registered [ 5335.526367] Key type .llcrypt registered [ 5335.591272] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 5335.962537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5335.981556] alg: No test for adler32 (adler32-zlib) [ 5336.816857] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 5336.893407] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 5338.463200] Key type lgssc registered [ 5338.983488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5342.024846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5342.038255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5343.141175] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5343.147290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5343.210936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5344.849893] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5348.525240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5348.572843] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5348.587996] Lustre: 387049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5348.732780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5350.358288] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5352.718672] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5354.994800] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5357.537322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5357.538472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5357.544489] Lustre: Skipped 1 previous similar message [ 5362.655931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5362.658383] Lustre: Skipped 2 previous similar messages [ 5367.776922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5367.781406] Lustre: Skipped 1 previous similar message [ 5370.335108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5370.397616] Lustre: server umount lustre-MDT0000 complete [ 5372.112756] LustreError: 387044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786547008 with bad export cookie 705498502733721206 [ 5372.118380] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5372.120146] LustreError: 387044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5372.283510] Lustre: server umount lustre-MDT0001 complete [ 5376.430577] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:03:33 (1786547013) [ 5378.676614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5378.838740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5380.086338] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5384.160198] LustreError: 389565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5384.168293] LustreError: 389565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5384.191201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5384.309890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5385.868637] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5389.197813] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5391.781873] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5394.941048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5395.040477] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5395.141337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5397.693165] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5400.902074] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5405.227795] Lustre: Permanently deactivating lustre-OST0000 [ 5405.234745] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5407.090788] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5408.835520] LustreError: 392455:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5408.843405] LustreError: 392455:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 5408.848159] LustreError: 392455:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5409.636231] LustreError: 392503:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 5409.640632] LustreError: 392503:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5416.672069] Lustre: server umount lustre-OST0000 complete [ 5417.670765] LustreError: 392753:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 5417.675085] LustreError: 392753:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5420.512455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5420.517959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5420.527378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5423.074530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5428.192227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5428.194638] Lustre: Skipped 3 previous similar messages [ 5429.254693] Lustre: server umount lustre-MDT0000 complete [ 5430.509354] LustreError: 392256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786547067 with bad export cookie 705498502733721773 [ 5430.516637] LustreError: 392256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5430.516675] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5441.054615] Lustre: server umount lustre-MDT0001 complete [ 5443.941063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5445.218423] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5446.283998] LustreError: 393911:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5449.268858] Lustre: 394249:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.159@tcp [ 5450.256313] Lustre: server umount MGS complete [ 5455.404342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5455.628753] LustreError: 394972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5455.636479] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5455.641574] Lustre: Skipped 1 previous similar message [ 5455.669103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5456.977210] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5460.299150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5460.421223] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5462.082718] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5465.097594] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5467.667740] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5468.365374] Lustre: Permanently reactivating lustre-OST0000 [ 5468.371598] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5468.374182] Lustre: Skipped 2 previous similar messages [ 5492.703212] LustreError: 396705:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5496.396142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5496.579603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5496.585364] Lustre: Skipped 1 previous similar message [ 5498.842495] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5502.114195] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5518.308694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5518.310882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5518.312107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5518.316733] Lustre: Skipped 1 previous similar message [ 5518.327764] Lustre: Skipped 1 previous similar message [ 5519.949619] Lustre: server umount lustre-OST0000 complete [ 5521.887771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5521.889802] LustreError: Skipped 1 previous similar message [ 5523.425021] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5523.428244] Lustre: Skipped 1 previous similar message [ 5527.249608] Lustre: server umount lustre-MDT0000 complete [ 5528.392121] LustreError: 394952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786547165 with bad export cookie 705498502733722795 [ 5528.395068] LustreError: 394952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5528.395619] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5528.512750] Lustre: server umount lustre-MDT0001 complete [ 5531.779316] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 5532.679433] Key type lgssc unregistered [ 5532.792319] LNet: 399552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5532.794386] LNetError: 399552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5532.800262] LNet: Removed LNI 192.168.203.159@tcp [ 5533.069330] Key type .llcrypt unregistered [ 5533.070258] Key type ._llcrypt unregistered [ 5539.931072] Key type ._llcrypt registered [ 5539.932507] Key type .llcrypt registered [ 5539.966076] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 5546.337695] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 5546.634206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5546.668138] alg: No test for adler32 (adler32-zlib) [ 5547.514037] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 5547.598843] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 5549.191202] Key type lgssc registered [ 5549.667473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5562.684605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5562.695384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5563.797417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5563.809420] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5563.858740] Lustre: lustre-MDT0000: new disk, initializing [ 5563.881618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5563.888132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5565.629425] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5570.697130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5570.756989] Lustre: 402681:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5570.791784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5570.796179] Lustre: Skipped 1 previous similar message [ 5570.848223] Lustre: lustre-MDT0001: new disk, initializing [ 5570.894240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5570.912942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5570.919085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5572.413509] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5577.298927] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5579.980826] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5583.187672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5583.320881] Lustre: lustre-OST0000: new disk, initializing [ 5583.324570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5583.329791] Lustre: 404647:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5583.377034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5585.660712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5585.664030] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5585.704989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5585.909814] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5590.504065] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5592.477337] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5592.597931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5594.418293] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5594.487080] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5596.131058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5596.135475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5596.136364] LustreError: Skipped 1 previous similar message [ 5596.148879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5596.154535] Lustre: Skipped 1 previous similar message [ 5601.251370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5601.255743] Lustre: Skipped 1 previous similar message [ 5601.449149] Lustre: server umount lustre-OST0000 complete [ 5606.370288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5606.371799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5606.378735] Lustre: Skipped 2 previous similar messages [ 5606.385072] Lustre: Skipped 1 previous similar message [ 5609.392518] Lustre: server umount lustre-MDT0000 complete [ 5611.258152] LustreError: 403475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786547247 with bad export cookie 9341506259938171368 [ 5611.265552] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5611.266050] LustreError: 403475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5611.414362] Lustre: server umount lustre-MDT0001 complete [ 5614.776454] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5615.398297] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:07:32 (1786547252) [ 5618.496053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5618.721404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5620.094244] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5623.777873] LustreError: 407108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5623.789088] LustreError: 407108:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5624.237086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5624.441828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5626.267125] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5629.579497] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5632.137960] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5635.472167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5635.637538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5638.171651] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5641.663739] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5642.607111] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5649.311289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5659.104478] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5659.110382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5659.120174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5660.132348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5662.444463] Lustre: server umount lustre-OST0000 complete [ 5665.250806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5665.250866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5665.252617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5665.252628] Lustre: Skipped 2 previous similar messages [ 5665.277057] Lustre: Skipped 2 previous similar messages [ 5670.445748] Lustre: server umount lustre-MDT0000 complete [ 5671.811553] LustreError: 407088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786547308 with bad export cookie 9341506259938172250 [ 5671.816235] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5671.816819] LustreError: 407088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5671.945370] Lustre: server umount lustre-MDT0001 complete [ 5676.036719] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 5677.370667] Key type lgssc unregistered [ 5677.551702] LNet: 410851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5677.556987] LNetError: 410851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5677.571753] LNet: Removed LNI 192.168.203.159@tcp [ 5678.101102] Key type .llcrypt unregistered [ 5678.103639] Key type ._llcrypt unregistered [ 5687.245853] Key type ._llcrypt registered [ 5687.247192] Key type .llcrypt registered [ 5687.294445] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 5687.458544] Key type .llcrypt unregistered [ 5687.460802] Key type ._llcrypt unregistered [ 5694.441221] Key type ._llcrypt registered [ 5694.442545] Key type .llcrypt registered [ 5694.484574] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 5694.801307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5694.846063] alg: No test for adler32 (adler32-zlib) [ 5695.697042] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 5695.775088] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 5697.351208] Key type lgssc registered [ 5697.936441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5701.726871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5701.742987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5702.935591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5704.746016] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5708.800854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5708.930169] LustreError: 413041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5708.934970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5708.942978] LustreError: 413041:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5711.460762] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5713.058383] LustreError: 413044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5713.066344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5713.066933] LustreError: 413044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5714.775826] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5717.701830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5717.856313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5719.577770] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5728.225590] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5728.227782] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5728.230970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5728.234913] Lustre: Skipped 1 previous similar message [ 5728.241838] Lustre: Skipped 1 previous similar message [ 5733.346783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5733.351809] Lustre: Skipped 1 previous similar message [ 5733.528119] Lustre: server umount lustre-OST0000 complete [ 5737.955084] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5737.959897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5737.968344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5743.584747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5743.588175] Lustre: Skipped 3 previous similar messages [ 5748.705229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5748.710090] Lustre: Skipped 1 previous similar message [ 5749.215141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5749.317834] Lustre: server umount lustre-MDT0000 complete [ 5751.139762] LustreError: 413025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786547387 with bad export cookie 13212469671279441715 [ 5751.147376] LustreError: 413025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5751.148377] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5751.298542] Lustre: server umount lustre-MDT0001 complete [ 5755.942512] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 5757.388980] Key type lgssc unregistered [ 5757.560709] LNet: 416198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5757.565658] LNetError: 416198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5757.578537] LNet: Removed LNI 192.168.203.159@tcp [ 5758.081662] Key type .llcrypt unregistered [ 5758.084306] Key type ._llcrypt unregistered [ 5769.639780] Key type ._llcrypt registered [ 5769.640897] Key type .llcrypt registered [ 5769.690188] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 5770.034811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5770.068296] alg: No test for adler32 (adler32-zlib) [ 5770.922317] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 5771.018619] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 5772.607169] Key type lgssc registered [ 5773.204099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5776.966982] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5776.977832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5792.479530] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5797.599342] LustreError: 417565:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.159@tcp: failed processing log, type 4: rc = -110 [ 5824.223195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5826.933443] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5830.420309] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5833.738665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5833.992090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5834.030919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5835.874315] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5839.140843] LustreError: 418601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5839.150705] LustreError: 418601:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5840.021086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5840.188883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5841.982428] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5845.392509] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5848.048896] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5854.688467] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5854.693456] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5854.701998] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5855.714923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5855.718885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5855.718939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5856.870341] Lustre: server umount lustre-OST0000 complete [ 5860.834630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5860.836040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5860.837063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5860.837073] Lustre: Skipped 1 previous similar message [ 5860.843474] Lustre: Skipped 1 previous similar message [ 5864.888984] Lustre: server umount lustre-MDT0000 complete [ 5865.952949] LustreError: 418604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5865.963737] LustreError: 418604:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5866.674060] LustreError: 417572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786547503 with bad export cookie 8518693034254093285 [ 5866.680687] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5866.682278] LustreError: 417572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5866.847743] Lustre: server umount lustre-MDT0001 complete [ 5871.572585] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 5872.569713] Key type lgssc unregistered [ 5872.695394] LNet: 421137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5872.698119] LNetError: 421137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5872.706399] LNet: Removed LNI 192.168.203.159@tcp [ 5873.053726] Key type .llcrypt unregistered [ 5873.055040] Key type ._llcrypt unregistered [ 5885.470396] Key type ._llcrypt registered [ 5885.472489] Key type .llcrypt registered [ 5885.545205] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 5886.079157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5886.088665] alg: No test for adler32 (adler32-zlib) [ 5887.008496] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 5887.147563] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 5888.759218] Key type lgssc registered [ 5889.427721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5893.341321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5893.352272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5894.544851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5896.266239] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5900.526402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5900.717973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5902.591285] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5906.069309] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5908.783518] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5912.147124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5912.316336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5915.046805] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5918.514207] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5919.532378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5926.368360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5926.373260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5926.381836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5930.350223] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.59@tcp (stopping) [ 5930.979255] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5930.991304] Lustre: Skipped 1 previous similar message [ 5932.319386] Lustre: server umount lustre-MDT0000 complete [ 5933.175761] LustreError: 422512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.59@tcp arrived at 1786547569 with bad export cookie 14885172499310938132 [ 5933.180241] LustreError: 422512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5933.892422] LustreError: 425109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786547570 with bad export cookie 14885172499310937516 [ 5933.898251] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5936.096855] LustreError: 422529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5936.098580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5936.099167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5936.100137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5936.100145] Lustre: Skipped 3 previous similar messages [ 5936.110549] LustreError: 422529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5938.144440] LustreError: 423269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5938.145041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5938.154317] LustreError: 423269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5940.007722] Lustre: server umount lustre-OST0000 complete [ 5943.282416] Lustre: server umount lustre-MDT0001 complete [ 5948.183990] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 5949.700862] Key type lgssc unregistered [ 5949.878588] LNet: 426215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5949.883615] LNetError: 426215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5949.892608] LNet: Removed LNI 192.168.203.159@tcp [ 5950.355283] Key type .llcrypt unregistered [ 5950.356731] Key type ._llcrypt unregistered [ 5962.587185] Key type ._llcrypt registered [ 5962.588516] Key type .llcrypt registered [ 5962.638675] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 5962.954897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5963.002053] alg: No test for adler32 (adler32-zlib) [ 5963.874936] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 5963.983801] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 5965.583161] Key type lgssc registered [ 5966.147915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5969.912433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5969.925704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5971.099993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5972.826458] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5976.923564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5977.104383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5978.956986] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5982.230529] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5984.989196] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5988.403420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5988.567639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5991.155034] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5994.750756] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5995.757080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6006.624616] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.59@tcp (stopping) [ 6006.752333] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6006.757286] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6007.268080] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6007.269724] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6007.272797] Lustre: Skipped 1 previous similar message [ 6008.753634] Lustre: server umount lustre-MDT0001 complete [ 6011.360365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6011.367340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6011.376470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6011.381140] Lustre: Skipped 1 previous similar message [ 6012.385132] LustreError: 427609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6012.398796] LustreError: 427609:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6017.258578] Lustre: server umount lustre-OST0000 complete [ 6017.505784] LustreError: 427608:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6019.254190] Lustre: server umount lustre-MDT0000 complete [ 6025.731337] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 6027.171133] Key type lgssc unregistered [ 6027.329773] LNet: 431298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6027.334901] LNetError: 431298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6027.347638] LNet: Removed LNI 192.168.203.159@tcp [ 6027.805629] Key type .llcrypt unregistered [ 6027.807885] Key type ._llcrypt unregistered [ 6039.090670] Key type ._llcrypt registered [ 6039.091874] Key type .llcrypt registered [ 6039.136406] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 6045.931487] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 6046.241210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6046.270690] alg: No test for adler32 (adler32-zlib) [ 6047.140964] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 6047.243818] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 6048.831156] Key type lgssc registered [ 6049.338049] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6066.220263] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 6073.544088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6073.558274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6074.663316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6074.676883] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6074.718074] Lustre: lustre-MDT0000: new disk, initializing [ 6074.743580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6074.751890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6076.297126] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6084.045667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6084.181755] Lustre: lustre-OST0000: new disk, initializing [ 6084.183669] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6084.185638] Lustre: Skipped 1 previous similar message [ 6084.188956] Lustre: 436302:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6084.216560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6086.634293] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6091.482236] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6092.304743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6092.310837] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6092.323833] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6103.740157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6103.815639] Lustre: 435284:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6103.862309] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6103.912665] Lustre: lustre-MDT0001: new disk, initializing [ 6103.975668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6103.999100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6104.005610] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6105.694096] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6110.205737] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6110.274787] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6111.617502] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6113.734410] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 2 sec [ 6117.858825] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6117.859785] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6117.862565] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6117.866543] Lustre: Skipped 1 previous similar message [ 6117.870891] Lustre: Skipped 1 previous similar message [ 6121.453754] Lustre: server umount lustre-MDT0001 complete [ 6128.100770] LustreError: 435290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6128.113699] LustreError: 435290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6133.220491] LustreError: 435293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6133.231778] LustreError: 435293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6138.339351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6138.339771] LustreError: 437640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6138.349070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6138.349414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6138.349422] Lustre: Skipped 1 previous similar message [ 6140.519515] Lustre: server umount lustre-OST0000 complete [ 6148.624760] Lustre: server umount lustre-MDT0000 complete [ 6154.721951] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 6156.201299] Key type lgssc unregistered [ 6156.376631] LNet: 440064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6156.381395] LNetError: 440064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6156.392572] LNet: Removed LNI 192.168.203.159@tcp [ 6156.834714] Key type .llcrypt unregistered [ 6156.837054] Key type ._llcrypt unregistered [ 6171.916653] Key type ._llcrypt registered [ 6171.918385] Key type .llcrypt registered [ 6171.981930] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 6172.146353] Key type .llcrypt unregistered [ 6172.148367] Key type ._llcrypt unregistered [ 6179.977559] Key type ._llcrypt registered [ 6179.979031] Key type .llcrypt registered [ 6180.037714] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 6180.409981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6180.465287] alg: No test for adler32 (adler32-zlib) [ 6181.319973] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 6181.406747] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 6182.991171] Key type lgssc registered [ 6183.567332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6195.457721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6195.465604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6196.619719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6196.632710] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6196.676657] Lustre: lustre-MDT0000: new disk, initializing [ 6196.700689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6196.706796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6198.384247] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6203.708875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6203.781430] Lustre: 443415:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6203.829228] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6203.834977] Lustre: Skipped 1 previous similar message [ 6203.881819] Lustre: lustre-MDT0001: new disk, initializing [ 6203.921492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6203.934654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6203.937652] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6205.747850] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6210.530304] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6212.936979] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6215.963685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6216.102957] Lustre: lustre-OST0000: new disk, initializing [ 6216.105258] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6216.108497] Lustre: 445383:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6216.139355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6218.555357] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6222.347104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6222.358808] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6222.413604] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6222.572356] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6227.428703] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6227.430951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6227.433861] Lustre: Skipped 1 previous similar message [ 6227.439014] Lustre: Skipped 2 previous similar messages [ 6231.037085] Lustre: server umount lustre-MDT0000 complete [ 6232.250869] LustreError: 444210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786547868 with bad export cookie 9099856078972932802 [ 6232.253857] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6232.254296] LustreError: 444210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6232.397945] Lustre: server umount lustre-MDT0001 complete [ 6244.104732] Lustre: server umount lustre-OST0000 complete [ 6248.008126] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 11:18:04 (1786547884) [ 6250.935321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6251.225792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6253.146779] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6256.608413] LustreError: 447410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6256.612212] LustreError: 447410:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6256.692372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6256.808879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6258.454314] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6261.795784] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6264.303454] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6280.548595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6280.663079] Lustre: Found index 0 for lustre-OST0000, updating log [ 6280.735183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6282.928716] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6286.348695] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6297.130116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6307.299225] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6307.299862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6307.304198] LustreError: Skipped 1 previous similar message [ 6307.311786] Lustre: Skipped 1 previous similar message [ 6307.315993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6307.320941] Lustre: Skipped 1 previous similar message [ 6308.387749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6308.391797] Lustre: Skipped 1 previous similar message [ 6309.607630] Lustre: server umount lustre-OST0000 complete [ 6322.723280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6322.724566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6322.729705] Lustre: Skipped 1 previous similar message [ 6322.735087] Lustre: Skipped 1 previous similar message [ 6327.777213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6327.781535] Lustre: Skipped 1 previous similar message [ 6335.969265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6335.973443] Lustre: Skipped 5 previous similar messages [ 6336.991156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6337.085855] Lustre: server umount lustre-MDT0000 complete [ 6338.846111] LustreError: 447389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786547975 with bad export cookie 9099856078972934181 [ 6338.852419] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6338.855516] LustreError: 447389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6339.003695] Lustre: server umount lustre-MDT0001 complete [ 6342.175297] LNet: 1 local NIs in recovery (showing 1): 192.168.203.159@tcp [ 6342.614223] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 11:19:39 (1786547979) [ 6926.951142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6927.167802] LustreError: 453838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6927.204242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6928.746881] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6932.448639] LustreError: 453839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6932.813347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6932.975599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6934.804268] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6937.597907] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6939.945346] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6956.196952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6956.317987] Lustre: Found index 0 for lustre-OST0000, updating log [ 6956.424872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6958.784205] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6962.197227] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6964.709762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6964.709816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6964.724158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6965.209895] Lustre: server umount lustre-OST0000 complete [ 6967.970445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6967.972466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6967.978157] Lustre: Skipped 1 previous similar message [ 6967.984671] Lustre: Skipped 1 previous similar message [ 6972.065546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6972.069582] Lustre: Skipped 7 previous similar messages [ 6973.177684] Lustre: server umount lustre-MDT0000 complete [ 6974.114795] LustreError: 454577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6974.125225] LustreError: 454577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6974.942836] LustreError: 453820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786548611 with bad export cookie 9099856078972935252 [ 6974.948245] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6974.949857] LustreError: 453820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6975.091810] Lustre: server umount lustre-MDT0001 complete [ 6982.175315] LNet: 1 local NIs in recovery (showing 1): 192.168.203.159@tcp [ 6994.867313] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 6999.140225] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 7015.496196] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 11:30:51 (1786548651) [ 7023.234535] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 7024.629218] Key type lgssc unregistered [ 7024.788720] LNet: 460496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7024.793710] LNetError: 460496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7087.269438] LNet: Removed LNI 192.168.203.159@tcp [ 7087.701804] Key type .llcrypt unregistered [ 7087.703982] Key type ._llcrypt unregistered [ 7096.401953] Key type ._llcrypt registered [ 7096.403826] Key type .llcrypt registered [ 7096.469238] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 7096.782682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7096.788769] alg: No test for adler32 (adler32-zlib) [ 7097.693495] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 7097.830732] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 7099.447147] Key type lgssc registered [ 7100.030114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7102.182733] LNet: Added LNI 192.168.203.159@tcp100 [8/256/0/180] [ 7102.589950] LNet: Added LNI 192.168.203.159@tcp101 [8/256/0/180] [ 7103.011385] LNet: Added LNI 192.168.203.159@tcp102 [8/256/0/180] [ 7103.450762] LNet: Added LNI 192.168.203.159@tcp103 [8/256/0/180] [ 7103.887586] LNet: Added LNI 192.168.203.159@tcp104 [8/256/0/180] [ 7104.315694] LNet: Added LNI 192.168.203.159@tcp105 [8/256/0/180] [ 7104.726950] LNet: Added LNI 192.168.203.159@tcp106 [8/256/0/180] [ 7105.133496] LNet: Added LNI 192.168.203.159@tcp107 [8/256/0/180] [ 7105.570127] LNet: Added LNI 192.168.203.159@tcp108 [8/256/0/180] [ 7106.005308] LNet: Added LNI 192.168.203.159@tcp109 [8/256/0/180] [ 7106.423328] LNet: Added LNI 192.168.203.159@tcp110 [8/256/0/180] [ 7106.847228] LNet: Added LNI 192.168.203.159@tcp111 [8/256/0/180] [ 7107.276554] LNet: Added LNI 192.168.203.159@tcp112 [8/256/0/180] [ 7107.704540] LNet: Added LNI 192.168.203.159@tcp113 [8/256/0/180] [ 7108.139085] LNet: Added LNI 192.168.203.159@tcp114 [8/256/0/180] [ 7108.571249] LNet: Added LNI 192.168.203.159@tcp115 [8/256/0/180] [ 7108.998306] LNet: Added LNI 192.168.203.159@tcp116 [8/256/0/180] [ 7109.431189] LNet: Added LNI 192.168.203.159@tcp117 [8/256/0/180] [ 7109.865179] LNet: Added LNI 192.168.203.159@tcp118 [8/256/0/180] [ 7110.299789] LNet: Added LNI 192.168.203.159@tcp119 [8/256/0/180] [ 7110.727424] LNet: Added LNI 192.168.203.159@tcp120 [8/256/0/180] [ 7111.156940] LNet: Added LNI 192.168.203.159@tcp121 [8/256/0/180] [ 7111.587496] LNet: Added LNI 192.168.203.159@tcp122 [8/256/0/180] [ 7112.008852] LNet: Added LNI 192.168.203.159@tcp123 [8/256/0/180] [ 7112.436237] LNet: Added LNI 192.168.203.159@tcp124 [8/256/0/180] [ 7112.859104] LNet: Added LNI 192.168.203.159@tcp125 [8/256/0/180] [ 7113.277057] LNet: Added LNI 192.168.203.159@tcp126 [8/256/0/180] [ 7113.707347] LNet: Added LNI 192.168.203.159@tcp127 [8/256/0/180] [ 7114.139167] LNet: Added LNI 192.168.203.159@tcp128 [8/256/0/180] [ 7114.571490] LNet: Added LNI 192.168.203.159@tcp129 [8/256/0/180] [ 7114.989240] LNet: Added LNI 192.168.203.159@tcp130 [8/256/0/180] [ 7115.391690] LNet: Added LNI 192.168.203.159@tcp131 [8/256/0/180] [ 7115.820948] LNet: Added LNI 192.168.203.159@tcp132 [8/256/0/180] [ 7116.239874] LNet: Added LNI 192.168.203.159@tcp133 [8/256/0/180] [ 7116.671625] LNet: Added LNI 192.168.203.159@tcp134 [8/256/0/180] [ 7117.104187] LNet: Added LNI 192.168.203.159@tcp135 [8/256/0/180] [ 7120.267233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7120.280431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7121.330982] Lustre: 463492:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.159@tcp: can't add NID 192.168.203.159@tcp130: rc = -75 [ 7121.364515] LNetError: 463512:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.159 [ 7121.369698] LNetError: Protocol error connecting to 192.168.203.159@tcp124 at host 192.168.203.159:988: is it running a compatible version of Lustre? [ 7121.375109] LNet: 1 local NIs in recovery (showing 1): 192.168.203.159@tcp123 [ 7121.375419] LNetError: Skipped 6 previous similar messages [ 7121.414644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7121.424426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7121.460586] Lustre: lustre-MDT0000: new disk, initializing [ 7121.488539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7121.497137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7122.404686] LNetError: 463514:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.159 [ 7122.414888] LNetError: 463514:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [ 7122.425862] LNetError: Protocol error connecting to 192.168.203.159@tcp103 at host 192.168.203.159:988: is it running a compatible version of Lustre? [ 7122.432079] LNetError: Skipped 5 previous similar messages [ 7123.355981] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7124.450842] LNetError: 463512:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.159 [ 7124.457322] LNetError: 463512:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 7124.462452] LNetError: Protocol error connecting to 192.168.203.159@tcp123 at host 192.168.203.159:988: is it running a compatible version of Lustre? [ 7128.546187] LNetError: 460957:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.159 [ 7128.546692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7128.556265] LNetError: Protocol error connecting to 192.168.203.159@tcp123 at host 192.168.203.159:988: is it running a compatible version of Lustre? [ 7128.622917] Lustre: 463509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7128.659461] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7128.662043] Lustre: Skipped 1 previous similar message [ 7128.701194] Lustre: lustre-MDT0001: new disk, initializing [ 7128.713864] Lustre: 464311:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.159@tcp: can't add NID 192.168.203.159@tcp130: rc = -75 [ 7128.717400] Lustre: 464311:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7128.771823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7128.789136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7128.794038] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7129.986136] Lustre: 464476:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.159@tcp: can't add NID 192.168.203.159@tcp130: rc = -75 [ 7129.992828] Lustre: 464476:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7130.453414] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7135.377163] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7136.737124] LNetError: 463521:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.159 [ 7136.743916] LNetError: Protocol error connecting to 192.168.203.159@tcp123 at host 192.168.203.159:988: is it running a compatible version of Lustre? [ 7137.926550] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7141.048115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7141.201946] Lustre: lustre-OST0000: new disk, initializing [ 7141.204804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7141.209078] Lustre: 465485:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7141.258433] Lustre: 465490:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.159@tcp: can't add NID 192.168.203.159@tcp130: rc = -75 [ 7141.262952] Lustre: 465490:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 7141.284220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7143.055940] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7147.547775] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7148.524882] Lustre: 466293:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.159@tcp: can't add NID 192.168.203.159@tcp130: rc = -75 [ 7148.527707] Lustre: 466293:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7148.543085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7148.545851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7148.582840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7151.071855] LNetError: 463517:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.159 [ 7151.074709] LNetError: Protocol error connecting to 192.168.203.159@tcp128 at host 192.168.203.159:988: is it running a compatible version of Lustre? [ 7153.633750] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7153.634363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7153.636324] LustreError: Skipped 1 previous similar message [ 7153.636753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7153.639565] Lustre: Skipped 1 previous similar message [ 7153.643678] Lustre: Skipped 1 previous similar message [ 7155.592402] Lustre: server umount lustre-OST0000 complete [ 7158.754455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7158.755763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7158.762404] Lustre: Skipped 1 previous similar message [ 7163.872935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7163.877651] Lustre: Skipped 2 previous similar messages [ 7168.480574] LNetError: 463514:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.159 [ 7168.487479] LNetError: 463514:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [ 7168.492889] LNetError: Protocol error connecting to 192.168.203.159@tcp103 at host 192.168.203.159:988: is it running a compatible version of Lustre? [ 7168.500060] LNetError: Skipped 11 previous similar messages [ 7168.992925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7168.997469] Lustre: Skipped 1 previous similar message [ 7171.039234] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7171.126464] Lustre: server umount lustre-MDT0000 complete [ 7172.866102] LustreError: 463500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786548809 with bad export cookie 17586516187160839932 [ 7172.873804] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7172.876329] LustreError: 463500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7173.039702] Lustre: server umount lustre-MDT0001 complete [ 7177.653420] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 7179.122721] Key type lgssc unregistered [ 7179.340959] LNet: 467398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7179.347460] LNetError: 467398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7179.353332] LNet: Removed LNI 192.168.203.159@tcp [ 7180.383205] LNet: Removed LNI 192.168.203.159@tcp100 [ 7180.385815] LNetError: 467398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7180.391155] LNetError: 467398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7180.396418] LNet: Removed LNI 192.168.203.159@tcp101 [ 7181.407202] LNet: Removed LNI 192.168.203.159@tcp102 [ 7181.409932] LNetError: 467398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7181.414925] LNetError: 467398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7181.420525] LNet: Removed LNI 192.168.203.159@tcp103 [ 7182.431200] LNet: Removed LNI 192.168.203.159@tcp104 [ 7183.455199] LNet: Removed LNI 192.168.203.159@tcp105 [ 7183.457815] LNetError: 467398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7183.463031] LNetError: 467398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7183.468439] LNet: Removed LNI 192.168.203.159@tcp106 [ 7184.479265] LNet: Removed LNI 192.168.203.159@tcp107 [ 7185.503202] LNet: Removed LNI 192.168.203.159@tcp108 [ 7186.527189] LNet: Removed LNI 192.168.203.159@tcp109 [ 7187.551180] LNet: Removed LNI 192.168.203.159@tcp110 [ 7187.553934] LNetError: 467398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7187.558625] LNetError: 467398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7187.563686] LNet: Removed LNI 192.168.203.159@tcp111 [ 7188.575199] LNet: Removed LNI 192.168.203.159@tcp112 [ 7189.599192] LNet: Removed LNI 192.168.203.159@tcp113 [ 7190.623189] LNet: Removed LNI 192.168.203.159@tcp114 [ 7191.647152] LNet: Removed LNI 192.168.203.159@tcp115 [ 7192.671135] LNet: Removed LNI 192.168.203.159@tcp116 [ 7193.695148] LNet: Removed LNI 192.168.203.159@tcp117 [ 7194.719250] LNet: Removed LNI 192.168.203.159@tcp118 [ 7195.743209] LNet: Removed LNI 192.168.203.159@tcp119 [ 7195.746042] LNetError: 467398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7195.751188] LNetError: 467398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7195.756547] LNet: Removed LNI 192.168.203.159@tcp120 [ 7196.767152] LNet: Removed LNI 192.168.203.159@tcp121 [ 7197.791132] LNet: Removed LNI 192.168.203.159@tcp122 [ 7198.815198] LNet: Removed LNI 192.168.203.159@tcp123 [ 7199.839136] LNet: Removed LNI 192.168.203.159@tcp124 [ 7200.863139] LNet: Removed LNI 192.168.203.159@tcp125 [ 7201.887148] LNet: Removed LNI 192.168.203.159@tcp126 [ 7202.911205] LNet: Removed LNI 192.168.203.159@tcp127 [ 7203.935191] LNet: Removed LNI 192.168.203.159@tcp128 [ 7204.959165] LNet: Removed LNI 192.168.203.159@tcp129 [ 7204.961827] LNet: Removed LNI 192.168.203.159@tcp130 [ 7204.964370] LNet: Removed LNI 192.168.203.159@tcp131 [ 7204.966877] LNet: Removed LNI 192.168.203.159@tcp132 [ 7204.969426] LNet: Removed LNI 192.168.203.159@tcp133 [ 7204.972038] LNet: Removed LNI 192.168.203.159@tcp134 [ 7204.980863] LNet: Removed LNI 192.168.203.159@tcp135 [ 7205.402549] Key type .llcrypt unregistered [ 7205.404681] Key type ._llcrypt unregistered [ 7215.195187] Key type ._llcrypt registered [ 7215.196396] Key type .llcrypt registered [ 7215.239141] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 7222.667410] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 7223.026771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7223.065376] alg: No test for adler32 (adler32-zlib) [ 7223.930139] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 7224.021360] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 7225.623136] Key type lgssc registered [ 7226.170686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7238.931168] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 11:34:35 (1786548875) [ 7245.930457] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 7247.264122] Key type lgssc unregistered [ 7247.403481] LNet: 471461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7247.405797] LNetError: 471461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7247.416388] LNet: Removed LNI 192.168.203.159@tcp [ 7247.746410] Key type .llcrypt unregistered [ 7247.749588] Key type ._llcrypt unregistered [ 7256.322679] Key type ._llcrypt registered [ 7256.324809] Key type .llcrypt registered [ 7256.393557] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 7256.858400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7256.899780] alg: No test for adler32 (adler32-zlib) [ 7257.750798] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 7257.848675] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 7259.431165] Key type lgssc registered [ 7259.990484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7264.610640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7264.627997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7265.722597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7265.738784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7265.785811] Lustre: lustre-MDT0000: new disk, initializing [ 7265.808906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7265.815717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7267.464091] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7272.676655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7272.749130] Lustre: 472652:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7272.787049] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7272.790866] Lustre: Skipped 1 previous similar message [ 7272.834984] Lustre: lustre-MDT0001: new disk, initializing [ 7272.869413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7272.885374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7272.889858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7274.167194] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7278.922197] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7281.453486] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7284.729491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7284.890979] Lustre: lustre-OST0000: new disk, initializing [ 7284.895537] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7284.901472] Lustre: 474620:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7284.944473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7286.843594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7286.848794] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7286.885634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7287.521565] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7292.375215] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7294.104526] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing lnet_if_list [ 7296.351475] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7296.370962] LNet: Added LNI 192.168.203.159@tcp10 [8/256/0/180] [ 7298.917732] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7298.938606] LNet: Added LNI 192.168.203.159@tcp11 [8/256/0/180] [ 7300.714319] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7300.733883] LNet: Added LNI 192.168.203.159@tcp12 [8/256/0/180] [ 7302.674968] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7302.693810] LNet: Added LNI 192.168.203.159@tcp13 [8/256/0/180] [ 7304.588184] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7304.605546] LNet: Added LNI 192.168.203.159@tcp14 [8/256/0/180] [ 7306.642473] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7306.659312] LNet: Added LNI 192.168.203.159@tcp15 [8/256/0/180] [ 7308.322369] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7308.334479] LNet: Added LNI 192.168.203.159@tcp16 [8/256/0/180] [ 7310.194308] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7310.208669] LNet: Added LNI 192.168.203.159@tcp17 [8/256/0/180] [ 7312.289243] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7312.308640] LNet: Added LNI 192.168.203.159@tcp18 [8/256/0/180] [ 7314.377254] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7314.394714] LNet: Added LNI 192.168.203.159@tcp19 [8/256/0/180] [ 7316.622045] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7316.641040] LNet: Added LNI 192.168.203.159@tcp20 [8/256/0/180] [ 7318.988482] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7319.004583] LNet: Added LNI 192.168.203.159@tcp21 [8/256/0/180] [ 7321.272735] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7321.287678] LNet: Added LNI 192.168.203.159@tcp22 [8/256/0/180] [ 7323.363621] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7323.381949] LNet: Added LNI 192.168.203.159@tcp23 [8/256/0/180] [ 7325.803682] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7325.819769] LNet: Added LNI 192.168.203.159@tcp24 [8/256/0/180] [ 7328.383791] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7328.406250] LNet: Added LNI 192.168.203.159@tcp25 [8/256/0/180] [ 7330.825738] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7330.837793] LNet: Added LNI 192.168.203.159@tcp26 [8/256/0/180] [ 7333.336325] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7333.358782] LNet: Added LNI 192.168.203.159@tcp27 [8/256/0/180] [ 7336.121611] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7336.135305] LNet: Added LNI 192.168.203.159@tcp28 [8/256/0/180] [ 7339.000587] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7339.020812] LNet: Added LNI 192.168.203.159@tcp29 [8/256/0/180] [ 7342.129452] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7342.148434] LNet: Added LNI 192.168.203.159@tcp30 [8/256/0/180] [ 7345.167368] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7345.188585] LNet: Added LNI 192.168.203.159@tcp31 [8/256/0/180] [ 7348.537358] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7348.557334] LNet: Added LNI 192.168.203.159@tcp32 [8/256/0/180] [ 7351.785291] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7351.805092] LNet: Added LNI 192.168.203.159@tcp33 [8/256/0/180] [ 7354.984000] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7355.002585] LNet: Added LNI 192.168.203.159@tcp34 [8/256/0/180] [ 7357.852991] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7357.867861] LNet: Added LNI 192.168.203.159@tcp35 [8/256/0/180] [ 7361.263347] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7361.282686] LNet: Added LNI 192.168.203.159@tcp36 [8/256/0/180] [ 7364.746033] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7364.763916] LNet: Added LNI 192.168.203.159@tcp37 [8/256/0/180] [ 7367.865326] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7367.884521] LNet: Added LNI 192.168.203.159@tcp38 [8/256/0/180] [ 7371.165995] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7371.187154] LNet: Added LNI 192.168.203.159@tcp39 [8/256/0/180] [ 7374.027397] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7374.047250] LNet: Added LNI 192.168.203.159@tcp40 [8/256/0/180] [ 7377.576111] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7377.589450] LNet: Added LNI 192.168.203.159@tcp41 [8/256/0/180] [ 7380.035701] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7380.045295] LNet: Added LNI 192.168.203.159@tcp42 [8/256/0/180] [ 7382.428656] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7382.437873] LNet: Added LNI 192.168.203.159@tcp43 [8/256/0/180] [ 7384.661546] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7384.670844] LNet: Added LNI 192.168.203.159@tcp44 [8/256/0/180] [ 7387.264620] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7387.274401] LNet: Added LNI 192.168.203.159@tcp45 [8/256/0/180] [ 7389.887785] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7389.897631] LNet: Added LNI 192.168.203.159@tcp46 [8/256/0/180] [ 7392.594030] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7392.603739] LNet: Added LNI 192.168.203.159@tcp47 [8/256/0/180] [ 7395.323542] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7395.333476] LNet: Added LNI 192.168.203.159@tcp48 [8/256/0/180] [ 7398.025252] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7398.035026] LNet: Added LNI 192.168.203.159@tcp49 [8/256/0/180] [ 7400.854114] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7400.867854] LNet: Added LNI 192.168.203.159@tcp50 [8/256/0/180] [ 7409.634326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7409.636690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7409.637374] Lustre: Skipped 2 previous similar messages [ 7409.641056] Lustre: Skipped 1 previous similar message [ 7414.754408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7414.758247] Lustre: Skipped 2 previous similar messages [ 7415.211817] Lustre: server umount lustre-MDT0000 complete [ 7416.479518] LustreError: 472646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786549053 with bad export cookie 486243267260417192 [ 7416.485420] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7416.618915] Lustre: server umount lustre-MDT0001 complete [ 7425.258884] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 7428.725993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7436.255171] Lustre: 471927:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786549056/real 1786549056] req@00000000d483d27e x1873332293168512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786549072 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7436.270573] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7438.303119] Lustre: 471927:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786549058/real 1786549058] req@00000000fcab4d7d x1873332293168640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786549074 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7440.351160] Lustre: 471924:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786549061/real 1786549061] req@00000000969d7fa9 x1873332293168768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786549077 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7441.375669] LustreError: 471923:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000aa0282ed x1873332293169536/t0(0) o250->MGC192.168.203.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7441.554857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7442.591142] Lustre: 471925:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786549063/real 1786549063] req@00000000d66496f6 x1873332293169024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786549079 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7443.494508] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7447.627328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7447.647179] Lustre: 471926:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786549068/real 1786549068] req@0000000066870667 x1873332293169152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786549084 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7447.802524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7449.363746] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7453.047576] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7453.157080] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7455.653033] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7464.420338] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7467.436222] Lustre: server umount lustre-OST0000 complete [ 7469.537360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7469.538367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7469.545712] Lustre: Skipped 1 previous similar message [ 7474.656990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7474.660899] Lustre: Skipped 2 previous similar messages [ 7478.791448] Lustre: server umount lustre-MDT0000 complete [ 7479.778797] LustreError: 498811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7479.789409] LustreError: 498811:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 7480.609299] LustreError: 472645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786549117 with bad export cookie 486243267260419299 [ 7480.617137] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7480.617150] LustreError: 472645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7491.124847] Lustre: server umount lustre-MDT0001 complete [ 7496.174490] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 7497.591204] Key type lgssc unregistered [ 7497.806764] LNet: 501589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7497.814663] LNetError: 501589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7497.821853] LNet: Removed LNI 192.168.203.159@tcp [ 7497.825489] LNet: Removed LNI 192.168.203.159@tcp10 [ 7497.829185] LNet: Removed LNI 192.168.203.159@tcp11 [ 7497.832690] LNet: Removed LNI 192.168.203.159@tcp12 [ 7497.836185] LNet: Removed LNI 192.168.203.159@tcp13 [ 7497.839387] LNet: Removed LNI 192.168.203.159@tcp14 [ 7497.842741] LNet: Removed LNI 192.168.203.159@tcp15 [ 7497.846195] LNet: Removed LNI 192.168.203.159@tcp16 [ 7497.849259] LNet: Removed LNI 192.168.203.159@tcp17 [ 7497.852887] LNet: Removed LNI 192.168.203.159@tcp18 [ 7497.855766] LNet: Removed LNI 192.168.203.159@tcp19 [ 7497.858720] LNet: Removed LNI 192.168.203.159@tcp20 [ 7497.861660] LNet: Removed LNI 192.168.203.159@tcp21 [ 7497.864585] LNet: Removed LNI 192.168.203.159@tcp22 [ 7497.867263] LNet: Removed LNI 192.168.203.159@tcp23 [ 7497.870068] LNet: Removed LNI 192.168.203.159@tcp24 [ 7497.872713] LNet: Removed LNI 192.168.203.159@tcp25 [ 7497.875338] LNet: Removed LNI 192.168.203.159@tcp26 [ 7497.877843] LNet: Removed LNI 192.168.203.159@tcp27 [ 7497.880375] LNet: Removed LNI 192.168.203.159@tcp28 [ 7497.882741] LNet: Removed LNI 192.168.203.159@tcp29 [ 7497.885144] LNet: Removed LNI 192.168.203.159@tcp30 [ 7497.887531] LNet: Removed LNI 192.168.203.159@tcp31 [ 7497.889811] LNet: Removed LNI 192.168.203.159@tcp32 [ 7497.891995] LNet: Removed LNI 192.168.203.159@tcp33 [ 7497.894252] LNet: Removed LNI 192.168.203.159@tcp34 [ 7497.896424] LNet: Removed LNI 192.168.203.159@tcp35 [ 7497.898536] LNet: Removed LNI 192.168.203.159@tcp36 [ 7497.900640] LNet: Removed LNI 192.168.203.159@tcp37 [ 7497.902731] LNet: Removed LNI 192.168.203.159@tcp38 [ 7497.904770] LNet: Removed LNI 192.168.203.159@tcp39 [ 7497.906730] LNet: Removed LNI 192.168.203.159@tcp40 [ 7497.908642] LNet: Removed LNI 192.168.203.159@tcp41 [ 7497.910440] LNet: Removed LNI 192.168.203.159@tcp42 [ 7497.912313] LNet: Removed LNI 192.168.203.159@tcp43 [ 7497.914181] LNet: Removed LNI 192.168.203.159@tcp44 [ 7497.916041] LNet: Removed LNI 192.168.203.159@tcp45 [ 7497.917747] LNet: Removed LNI 192.168.203.159@tcp46 [ 7497.919501] LNet: Removed LNI 192.168.203.159@tcp47 [ 7497.921262] LNet: Removed LNI 192.168.203.159@tcp48 [ 7497.922952] LNet: Removed LNI 192.168.203.159@tcp49 [ 7497.928282] LNet: Removed LNI 192.168.203.159@tcp50 [ 7498.321652] Key type .llcrypt unregistered [ 7498.324002] Key type ._llcrypt unregistered [ 7507.722450] Key type ._llcrypt registered [ 7507.723621] Key type .llcrypt registered [ 7507.784655] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 7514.181253] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 7514.582054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7514.635824] alg: No test for adler32 (adler32-zlib) [ 7515.508968] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 7515.603622] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 7517.207150] Key type lgssc registered [ 7517.660284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7531.594682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7531.606371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7532.755391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7532.772597] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7532.829185] Lustre: lustre-MDT0000: new disk, initializing [ 7532.876741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7532.887629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7534.963677] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7540.039380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7540.091218] Lustre: 504758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7540.118287] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7540.120557] Lustre: Skipped 1 previous similar message [ 7540.158493] Lustre: lustre-MDT0001: new disk, initializing [ 7540.194293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7540.208668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7540.213125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7541.450050] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7545.833231] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7547.699911] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7550.214979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7550.290653] Lustre: *** cfs_fail_loc=90f, val=0*** [ 7550.321130] Lustre: lustre-OST0000: new disk, initializing [ 7550.322968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7550.325467] Lustre: 506773:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7550.356303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7550.402641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7550.405441] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7550.450281] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7552.335382] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7556.020710] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7560.674078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7560.674639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7560.676666] LustreError: Skipped 1 previous similar message [ 7560.682318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7560.684562] Lustre: Skipped 1 previous similar message [ 7564.561591] Lustre: server umount lustre-OST0000 complete [ 7565.792794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7565.793313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7565.793493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7565.796706] Lustre: Skipped 2 previous similar messages [ 7570.913602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7570.917059] Lustre: Skipped 2 previous similar messages [ 7576.032600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7576.038043] Lustre: Skipped 1 previous similar message [ 7580.127135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7580.367912] Lustre: server umount lustre-MDT0000 complete [ 7582.841522] LustreError: 504750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786549219 with bad export cookie 12419268690941454474 [ 7582.845545] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7582.851440] LustreError: 504750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7583.075738] Lustre: server umount lustre-MDT0001 complete [ 7590.500994] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 7592.757503] Key type lgssc unregistered [ 7593.023732] LNet: 508681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7593.033468] LNetError: 508681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7594.101379] LNet: Removed LNI 192.168.203.159@tcp [ 7594.836333] Key type .llcrypt unregistered [ 7594.838844] Key type ._llcrypt unregistered [ 7605.001928] Key type ._llcrypt registered [ 7605.003703] Key type .llcrypt registered [ 7605.038250] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 7605.456027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7605.470247] alg: No test for adler32 (adler32-zlib) [ 7606.313565] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 7606.401488] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 7607.983144] Key type lgssc registered [ 7608.343329] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7610.921797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7610.928172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7612.064443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7613.359140] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7616.115740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7616.268515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7617.531413] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7619.674823] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7621.317525] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7623.426879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7623.529475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7625.222957] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7627.385704] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7641.505934] loop0: detected capacity change from 0 to 204800000 [ 7641.573452] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7641.576761] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7641.582079] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7643.495640] loop0: detected capacity change from 0 to 204800000 [ 7643.547344] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7643.550950] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7643.555133] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7645.168162] loop0: detected capacity change from 0 to 204800000 [ 7646.918157] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7646.958336] Lustre: fs15246-OST0000: new disk, initializing [ 7646.959827] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7646.962117] Lustre: 513543:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7646.981043] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7648.801564] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7652.540803] loop1: detected capacity change from 0 to 204800000 [ 7654.345560] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7654.384925] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7654.407384] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7654.443487] Lustre: fs15246-MDT0000: new disk, initializing [ 7654.485530] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7654.495231] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7654.540246] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7655.585250] Lustre: 514666:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000b545aef6 x1873332659059968/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786549308 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7655.894716] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7660.640651] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7660.643846] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7660.648192] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7665.635059] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7665.638923] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7668.899990] Lustre: server umount fs15246-MDT0000 complete [ 7670.756611] LustreError: 509923:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7670.762718] LustreError: 509923:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7673.356249] Lustre: server umount fs15246-OST0000 complete [ 7680.995224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7681.002270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7681.006924] Lustre: Skipped 1 previous similar message [ 7686.114459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7686.120202] Lustre: Skipped 1 previous similar message [ 7691.241775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7691.247424] Lustre: Skipped 1 previous similar message [ 7694.303146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7694.390695] Lustre: server umount lustre-OST0000 complete [ 7698.400495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7698.410109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7698.420598] Lustre: Skipped 1 previous similar message [ 7698.427187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7701.473834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7703.570878] Lustre: server umount lustre-MDT0000 complete [ 7706.593466] LustreError: 513581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7706.606732] LustreError: 513581:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7706.680191] LustreError: 509909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786549343 with bad export cookie 9081398374470703925 [ 7706.682894] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7706.691832] LustreError: 509909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7707.090795] Lustre: server umount lustre-MDT0001 complete [ 7713.046099] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 7714.356222] Key type lgssc unregistered [ 7714.510571] LNet: 517760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7714.514613] LNetError: 517760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7714.524699] LNet: Removed LNI 192.168.203.159@tcp [ 7714.939251] Key type .llcrypt unregistered [ 7714.940970] Key type ._llcrypt unregistered [ 7723.521488] Key type ._llcrypt registered [ 7723.523648] Key type .llcrypt registered [ 7723.577456] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 7724.088498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7724.114208] alg: No test for adler32 (adler32-zlib) [ 7725.000188] Lustre: Lustre: Build Version: 2.17.57_1_g9701683 [ 7725.125775] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 7726.727157] Key type lgssc registered [ 7727.171529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7741.908202] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 7745.695415] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 7760.273964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7760.286207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7761.389367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7761.401925] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7761.447183] Lustre: lustre-MDT0000: new disk, initializing [ 7761.480836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7761.491251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7763.073054] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7767.912582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7767.966064] Lustre: 522197:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7768.005542] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7768.009183] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7768.012692] Lustre: Skipped 1 previous similar message [ 7768.057513] Lustre: lustre-MDT0001: new disk, initializing [ 7768.099911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7768.114501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7768.117874] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7769.616231] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7773.700613] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7775.646635] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7778.326963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7778.444120] Lustre: lustre-OST0000: new disk, initializing [ 7778.446782] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7778.451299] Lustre: 524164:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7778.494333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7780.674296] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7784.484944] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7785.470145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7785.473289] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7785.510079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7785.976805] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7786.064882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7787.691789] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7787.782100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7790.563123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7790.563440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7790.565076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7790.568240] Lustre: Skipped 1 previous similar message [ 7790.576765] LustreError: Skipped 1 previous similar message [ 7794.593196] Lustre: server umount lustre-OST0000 complete [ 7798.751926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7798.755452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7798.759940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7798.762986] Lustre: Skipped 1 previous similar message [ 7800.800920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7800.801869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7802.270185] Lustre: server umount lustre-MDT0000 complete [ 7803.726814] LustreError: 522990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786549440 with bad export cookie 10395202779437261491 [ 7803.728960] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7803.731275] LustreError: 522990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7803.872276] Lustre: server umount lustre-MDT0001 complete [ 7811.919619] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 7815.541703] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 7826.331436] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7698 sec ========== 11:44:22 (1786549462) [ 7826.952991] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:44:23 (1786549463) === [ 7828.182111] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:44:24 (1786549464) === [ 7842.423622] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 7843.512618] Key type lgssc unregistered [ 7843.648310] LNet: 530799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7843.651075] LNetError: 530799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7843.657300] LNet: Removed LNI 192.168.203.159@tcp [ 7843.990665] Key type .llcrypt unregistered [ 7843.991788] Key type ._llcrypt unregistered