[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 455755650 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002420] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.006761] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008014] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009006] pid_max: default: 32768 minimum: 301 [ 0.010117] LSM: Security Framework initializing [ 0.012032] Yama: becoming mindful. [ 0.012853] SELinux: Initializing. [ 0.013059] *** VALIDATE selinux *** [ 0.020381] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024687] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025140] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026094] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027082] *** VALIDATE tmpfs *** [ 0.029269] *** VALIDATE proc *** [ 0.030144] *** VALIDATE cgroup *** [ 0.031005] *** VALIDATE cgroup2 *** [ 0.032168] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033121] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035028] Spectre V2 : User space: Vulnerable [ 0.036007] Speculative Store Bypass: Vulnerable [ 0.038707] debug: unmapping init [mem 0xffffffffb8e59000-0xffffffffb8e60fff] [ 0.040121] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041591] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042023] ... version: 2 [ 0.043011] ... bit width: 48 [ 0.044006] ... generic registers: 4 [ 0.045007] ... value mask: 0000ffffffffffff [ 0.046009] ... max period: 00007fffffffffff [ 0.047008] ... fixed-purpose events: 3 [ 0.048009] ... event mask: 000000070000000f [ 0.049294] rcu: Hierarchical SRCU implementation. [ 0.051196] smp: Bringing up secondary CPUs ... [ 0.052526] x86: Booting SMP configuration: [ 0.053025] .... node #0, CPUs: #1 #2 #3 [ 0.056623] smp: Brought up 1 node, 4 CPUs [ 0.058026] smpboot: Max logical packages: 1 [ 0.059011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.240021] node 0 deferred pages initialised in 180ms [ 0.243154] devtmpfs: initialized [ 0.245310] x86/mm: Memory block size: 128MB [ 0.247781] gcov: version magic: 0x41383552 [ 0.250243] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.255125] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.257237] pinctrl core: initialized pinctrl subsystem [ 0.259130] [ 0.259567] ************************************************************* [ 0.261010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.263013] ** ** [ 0.265010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.268016] ** ** [ 0.269007] ** This means that this kernel is built to expose internal ** [ 0.271013] ** IOMMU data structures, which may compromise security on ** [ 0.273011] ** your system. ** [ 0.275011] ** ** [ 0.277012] ** If you see this message and you are not debugging the ** [ 0.279011] ** kernel, report this immediately to your vendor! ** [ 0.281011] ** ** [ 0.283012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.286015] ************************************************************* [ 0.288614] NET: Registered protocol family 16 [ 0.290287] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.292052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.295056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.299008] cpuidle: using governor menu [ 0.300291] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.303094] PCI: Using configuration type 1 for base access [ 0.305117] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.312084] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.315032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.317178] cryptd: max_cpu_qlen set to 1000 [ 0.320239] ACPI: Added _OSI(Module Device) [ 0.321009] ACPI: Added _OSI(Processor Device) [ 0.322011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.323008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.328199] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.334415] ACPI: Interpreter enabled [ 0.335079] ACPI: PM: (supports S0 S3 S4 S5) [ 0.336007] ACPI: Using IOAPIC for interrupt routing [ 0.338079] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.341374] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.350000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.352038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.354016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.358076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.363521] acpiphp: Slot [2] registered [ 0.365141] acpiphp: Slot [5] registered [ 0.366090] acpiphp: Slot [6] registered [ 0.367106] acpiphp: Slot [7] registered [ 0.369157] acpiphp: Slot [8] registered [ 0.370104] acpiphp: Slot [9] registered [ 0.371129] acpiphp: Slot [10] registered [ 0.373091] acpiphp: Slot [3] registered [ 0.374143] acpiphp: Slot [4] registered [ 0.375067] acpiphp: Slot [11] registered [ 0.377062] acpiphp: Slot [12] registered [ 0.378087] acpiphp: Slot [13] registered [ 0.380094] acpiphp: Slot [14] registered [ 0.381090] acpiphp: Slot [15] registered [ 0.382082] acpiphp: Slot [16] registered [ 0.383133] acpiphp: Slot [17] registered [ 0.385076] acpiphp: Slot [18] registered [ 0.386102] acpiphp: Slot [19] registered [ 0.387095] acpiphp: Slot [20] registered [ 0.389086] acpiphp: Slot [21] registered [ 0.390080] acpiphp: Slot [22] registered [ 0.392124] acpiphp: Slot [23] registered [ 0.393091] acpiphp: Slot [24] registered [ 0.395095] acpiphp: Slot [25] registered [ 0.396117] acpiphp: Slot [26] registered [ 0.397079] acpiphp: Slot [27] registered [ 0.398067] acpiphp: Slot [28] registered [ 0.400098] acpiphp: Slot [29] registered [ 0.401095] acpiphp: Slot [30] registered [ 0.402087] acpiphp: Slot [31] registered [ 0.404111] PCI host bridge to bus 0000:00 [ 0.405021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.408030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.410029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.413027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.415020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.418023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.420172] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.422812] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.426184] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.434014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.439615] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.441017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.444016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.446015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.448576] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.452921] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.455037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.457721] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.461991] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.473014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.477010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.481823] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.490019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.496015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.517020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.526569] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.531009] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.536013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.549011] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.558860] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.563013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.567016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.586013] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.595000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.599935] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.607016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.622016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.632400] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.640018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.647015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.660012] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.669749] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.675011] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.681012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.699012] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.708448] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.711493] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.714359] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.717423] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.721305] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.726144] iommu: Default domain type: Passthrough [ 0.728361] SCSI subsystem initialized [ 0.729119] ACPI: bus type USB registered [ 0.731087] usbcore: registered new interface driver usbfs [ 0.733112] usbcore: registered new interface driver hub [ 0.734052] usbcore: registered new device driver usb [ 0.736157] pps_core: LinuxPPS API ver. 1 registered [ 0.738009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.742073] PTP clock support registered [ 0.744045] EDAC MC: Ver: 3.0.0 [ 0.745402] PCI: Using ACPI for IRQ routing [ 0.746649] NetLabel: Initializing [ 0.748024] NetLabel: domain hash size = 128 [ 0.749008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.751096] NetLabel: unlabeled traffic allowed by default [ 0.753130] vgaarb: loaded [ 0.754149] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.756012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.761319] clocksource: Switched to clocksource kvm-clock [ 0.844882] VFS: Disk quotas dquot_6.6.0 [ 0.846123] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.848417] *** VALIDATE ramfs *** [ 0.849195] *** VALIDATE hugetlbfs *** [ 0.850147] pnp: PnP ACPI init [ 0.852552] pnp: PnP ACPI: found 6 devices [ 0.867256] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.869712] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.871262] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.873237] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.875475] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.877904] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.880488] NET: Registered protocol family 2 [ 0.882304] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.885699] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.888451] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.893094] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.895946] TCP: Hash tables configured (established 65536 bind 65536) [ 0.897896] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.900424] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.902134] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.905276] NET: Registered protocol family 1 [ 0.907579] RPC: Registered named UNIX socket transport module. [ 0.909439] RPC: Registered udp transport module. [ 0.910849] RPC: Registered tcp transport module. [ 0.912274] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.913956] NET: Registered protocol family 44 [ 0.915039] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.916779] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.918293] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.920117] PCI: CLS 0 bytes, default 64 [ 0.921597] Unpacking initramfs... [ 2.199728] debug: unmapping init [mem 0xffff9e38fcc54000-0xffff9e38fffbffff] [ 2.203464] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.206156] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.208489] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.681565] Initialise system trusted keyrings [ 2.683504] Key type blacklist registered [ 2.685467] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.693952] zbud: loaded [ 2.696535] *** VALIDATE nfs *** [ 2.697516] *** VALIDATE nfs4 *** [ 2.698836] pstore: using deflate compression [ 2.701023] Platform Keyring initialized [ 2.784154] NET: Registered protocol family 38 [ 2.785886] Key type asymmetric registered [ 2.787414] Asymmetric key parser 'x509' registered [ 2.789382] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.792394] io scheduler mq-deadline registered [ 2.794407] io scheduler kyber registered [ 2.796281] io scheduler bfq registered [ 2.797444] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.799837] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.802239] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.805244] ACPI: Power Button [PWRF] [ 2.810533] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.816680] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.831767] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.837566] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.851344] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.878198] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.905909] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.910218] Non-volatile memory driver v1.3 [ 2.912210] Linux agpgart interface v0.103 [ 2.939115] virtio_blk virtio1: [vda] 139280 512-byte logical blocks (71.3 MB/68.0 MiB) [ 2.941061] vda: detected capacity change from 0 to 71311360 [ 2.956432] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.959517] vdb: detected capacity change from 0 to 1073741824 [ 2.975851] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.978950] vdc: detected capacity change from 0 to 2621440000 [ 2.992599] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.994828] vdd: detected capacity change from 0 to 2621440000 [ 3.006797] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.010086] vde: detected capacity change from 0 to 4294967296 [ 3.022433] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.024655] vdf: detected capacity change from 0 to 4294967296 [ 3.029307] libphy: Fixed MDIO Bus: probed [ 3.033627] usbcore: registered new interface driver usbserial_generic [ 3.035626] usbserial: USB Serial support registered for generic [ 3.038227] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.042101] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.043751] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.046243] mousedev: PS/2 mouse device common for all mice [ 3.048705] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.049482] rtc_cmos 00:05: RTC can wake from S4 [ 3.054238] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.055380] rtc_cmos 00:05: registered as rtc0 [ 3.058669] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.059274] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.065019] intel_pstate: CPU model not supported [ 3.067372] hid: raw HID events driver (C) Jiri Kosina [ 3.069109] usbcore: registered new interface driver usbhid [ 3.071450] usbhid: USB HID core driver [ 3.073162] drop_monitor: Initializing network drop monitor service [ 3.075768] Initializing XFRM netlink socket [ 3.077538] NET: Registered protocol family 10 [ 3.080435] Segment Routing with IPv6 [ 3.081823] NET: Registered protocol family 17 [ 3.084058] mpls_gso: MPLS GSO support [ 3.089662] RAS: Correctable Errors collector initialized. [ 3.091875] AVX version of gcm_enc/dec engaged. [ 3.093116] AES CTR mode by8 optimization enabled [ 3.153313] sched_clock: Marking stable (3153146251, 0)->(3970632949, -817486698) [ 3.156676] registered taskstats version 1 [ 3.158465] Loading compiled-in X.509 certificates [ 3.160313] zswap: loaded using pool lzo/zbud [ 3.180955] Key type big_key registered [ 3.191834] Key type encrypted registered [ 3.194106] ima: No TPM chip found, activating TPM-bypass! [ 3.196093] ima: Allocated hash algorithm: sha1 [ 3.197626] ima: No architecture policies found [ 3.198992] evm: Initialising EVM extended attributes: [ 3.200956] evm: security.selinux [ 3.202279] evm: security.ima [ 3.203554] evm: security.capability [ 3.204791] evm: HMAC attrs: 0x1 [ 3.207270] rtc_cmos 00:05: setting system clock to 2026-06-17 01:47:20 UTC (1781660840) [ 3.212471] debug: unmapping init [mem 0xffffffffb9e03000-0xffffffffb9ffffff] [ 3.215504] debug: unmapping init [mem 0xffffffffb8b82000-0xffffffffb8e58fff] [ 3.223133] Write protecting the kernel read-only data: 28672k [ 3.226478] debug: unmapping init [mem 0xffffffffb7203000-0xffffffffb73fffff] [ 3.229551] debug: unmapping init [mem 0xffffffffb7b14000-0xffffffffb7bfffff] [ 3.260265] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.267580] systemd[1]: Detected virtualization kvm. [ 3.269540] systemd[1]: Detected architecture x86-64. [ 3.271491] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.295432] systemd[1]: No hostname configured. [ 3.297165] systemd[1]: Set hostname to . [ 3.298692] random: systemd: uninitialized urandom read (16 bytes read) [ 3.300601] systemd[1]: Initializing machine ID from random generator. [ 3.346578] random: ln: uninitialized urandom read (6 bytes read) [ 3.420417] random: systemd: uninitialized urandom read (16 bytes read) [ 3.422777] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.428507] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ 3.431808] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.995441] device-mapper: uevent: version 1.0.3 [ 3.997268] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.659522] virtio_net virtio0 ens2: renamed from eth0 [ 4.702025] random: fast init done [ 4.844392] scsi host0: ata_piix [ 4.898559] scsi host1: ata_piix [ 4.899970] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.902206] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.040508] dracut-initqueue[590]: RTNETLINK answers: File exists [ 9.710066] random: crng init done [ 9.711647] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.969619] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.961181] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.183062] SELinux: Disabled at runtime. [ 11.241834] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.251209] systemd[1]: Detected virtualization kvm. [ 11.253205] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.645885] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.648221] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.651767] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.654923] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.656866] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.663909] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.668794] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ 11.765307] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.017621] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.290628] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.328798] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.450264] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.465129] EDAC sbridge: Ver: 1.1.2 [ 13.734183] Key type dns_resolver registered [ 14.014530] NFS: Registering the id_resolver key type [ 14.015914] Key type id_resolver registered [ 14.017059] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg436-server login: [ 36.253926] libcfs: loading out-of-tree module taints kernel. [ 36.272522] Key type ._llcrypt registered [ 36.274396] Key type .llcrypt registered [ 36.319864] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 44.247481] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 44.984987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 44.993076] alg: No test for adler32 (adler32-zlib) [ 45.974464] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 46.233071] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 47.847150] Key type lgssc registered [ 48.455269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 54.708294] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 67.745352] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 72.547812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 72.557808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 73.668231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 73.680307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 73.718693] Lustre: lustre-MDT0000: new disk, initializing [ 73.742972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 73.750612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 75.071862] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 80.506816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 80.543029] Lustre: 6479:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 80.559629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 80.562310] Lustre: Skipped 1 previous similar message [ 80.598369] Lustre: lustre-MDT0001: new disk, initializing [ 80.620790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 80.628735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 80.634136] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 81.968326] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 84.280714] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 87.303538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 87.390582] Lustre: lustre-OST0000: new disk, initializing [ 87.392598] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 87.395446] Lustre: 8418:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 87.414272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 89.218497] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 90.219033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 90.222536] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 90.237020] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 94.903760] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 94.951335] Lustre: lustre-OST0001: new disk, initializing [ 94.953475] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 94.956384] Lustre: 9489:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 94.976543] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 96.907472] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 98.477376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 98.480548] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 98.493455] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 102.511582] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 109.102223] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 110.945550] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing check_logdir /tmp/testlogs/ [ 112.383969] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing yml_node [ 113.641219] Lustre: DEBUG MARKER: Client: 2.17.54.40 [ 114.469655] Lustre: DEBUG MARKER: MDS: 2.17.54.40 [ 115.287903] Lustre: DEBUG MARKER: OSS: 2.17.54.40 [ 115.794764] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 16 21:49:12 EDT 2026 [ 121.406125] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 121.963574] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 134.623885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 134.624286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 134.626966] Lustre: Skipped 3 previous similar messages [ 134.629221] Lustre: Skipped 2 previous similar messages [ 139.743652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 139.745389] Lustre: Skipped 3 previous similar messages [ 139.864316] Lustre: server umount lustre-MDT0000 complete [ 141.064653] LustreError: 7897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781660978 with bad export cookie 3350959985612950693 [ 141.065790] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 141.069602] LustreError: 7897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 144.863719] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 144.864183] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 144.868021] Lustre: Skipped 1 previous similar message [ 144.872163] Lustre: Skipped 1 previous similar message [ 146.911723] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 146.914756] Lustre: Skipped 1 previous similar message [ 147.235301] Lustre: server umount lustre-MDT0001 complete [ 154.558490] Lustre: server umount lustre-OST0000 complete [ 161.867101] Lustre: server umount lustre-OST0001 complete [ 163.774105] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 166.889189] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 178.501338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.607798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 178.621962] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 178.669042] Lustre: lustre-MDT0000: new disk, initializing [ 178.698103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 178.706315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 180.083489] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 184.280607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.323597] Lustre: 15029:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 184.452384] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 185.932846] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 190.377762] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 202.843719] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 227.171375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 227.319783] Lustre: lustre-OST0000: new disk, initializing [ 227.322232] Lustre: Skipped 1 previous similar message [ 227.324462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 227.326200] Lustre: Skipped 2 previous similar messages [ 227.328498] Lustre: 17040:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 227.360932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 227.364474] Lustre: Skipped 1 previous similar message [ 229.373526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 229.376586] Lustre: Skipped 1 previous similar message [ 229.378800] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 229.427176] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 229.543985] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 233.481915] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 235.039601] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 235.119644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 236.639906] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 236.718415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 239.588383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 239.588448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 239.591743] LustreError: Skipped 1 previous similar message [ 239.599055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 239.602192] Lustre: Skipped 1 previous similar message [ 243.547536] Lustre: server umount lustre-OST0000 complete [ 246.239981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 246.243671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 246.249305] Lustre: Skipped 1 previous similar message [ 249.824155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 249.826589] Lustre: Skipped 2 previous similar messages [ 251.096984] Lustre: server umount lustre-MDT0000 complete [ 252.462205] LustreError: 15022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781661089 with bad export cookie 3350959985612952212 [ 252.465604] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 252.466469] LustreError: 15022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 252.593972] Lustre: server umount lustre-MDT0001 complete [ 253.977484] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 21:51:30 (1781661090) [ 254.887295] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 255.554819] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 21:51:31 (1781661091) [ 258.128870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 258.309471] LustreError: 19443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 258.316374] LustreError: 19443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 258.337391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 259.976459] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 263.528393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 265.372535] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 268.366250] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 270.287075] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 272.772461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 274.918507] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 277.509173] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 293.349668] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.349708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 293.360743] Lustre: Skipped 1 previous similar message [ 293.364149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 293.366799] Lustre: Skipped 2 previous similar messages [ 296.864631] Lustre: server umount lustre-OST0000 complete [ 304.480996] Lustre: server umount lustre-MDT0000 complete [ 305.770308] LustreError: 19425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781661143 with bad export cookie 3350959985612953094 [ 305.774221] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 305.775740] LustreError: 19425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 305.905790] Lustre: server umount lustre-MDT0001 complete [ 309.949262] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 311.134625] Key type lgssc unregistered [ 311.276455] LNet: 22994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 311.279727] LNetError: 22994:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 311.288455] LNet: Removed LNI 192.168.204.136@tcp [ 311.671131] Key type .llcrypt unregistered [ 311.673074] Key type ._llcrypt unregistered [ 320.501706] Key type ._llcrypt registered [ 320.502724] Key type .llcrypt registered [ 320.536920] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 320.877918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 320.928641] alg: No test for adler32 (adler32-zlib) [ 321.791445] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 321.873266] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 323.455168] Key type lgssc registered [ 323.827091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 326.536902] loop: module loaded [ 326.553329] loop0: detected capacity change from 0 to 204800000 [ 326.627173] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 326.630295] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 326.743541] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 328.680404] loop0: detected capacity change from 0 to 204800000 [ 328.801264] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 328.806253] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 328.811089] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 330.684784] loop0: detected capacity change from 0 to 204800000 [ 332.841893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 332.848534] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 333.953106] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 333.975120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 334.012981] Lustre: lustre-MDT0000: new disk, initializing [ 334.051152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 334.063054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 335.671898] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 339.495297] loop1: detected capacity change from 0 to 204800000 [ 341.443989] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 341.551588] Lustre: lustre-OST07c6: new disk, initializing [ 341.553575] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 341.555084] Lustre: Skipped 1 previous similar message [ 341.557339] Lustre: 26160:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 341.581574] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 343.046280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 343.050266] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 343.062951] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 343.617042] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 345.744294] Lustre: 25143:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 345.748996] Lustre: 25143:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 345.752286] Lustre: 25143:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 345.754911] Lustre: 25143:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 345.757752] Lustre: 25143:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 345.760168] Lustre: 25143:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 360.059525] Lustre: 25144:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 360.062873] Lustre: 25144:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 360.066620] Lustre: 25144:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 360.070134] Lustre: 25144:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 360.073891] Lustre: 25144:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 360.076638] Lustre: 25144:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 363.604768] Lustre: 25145:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 363.607555] Lustre: 25145:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 363.609597] Lustre: 25145:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 363.611948] Lustre: 25145:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 363.615450] Lustre: 25145:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 363.617604] Lustre: 25145:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 363.620930] Lustre: 25145:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 363.623678] Lustre: 25145:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 363.626054] Lustre: 25145:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 363.627917] Lustre: 25145:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 363.629749] Lustre: 25145:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 363.631791] Lustre: 25145:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 366.683264] Lustre: 25143:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 366.687869] Lustre: 25143:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 366.691149] Lustre: 25143:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 366.693986] Lustre: 25143:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 366.696515] Lustre: 25143:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 366.699473] Lustre: 25143:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 366.702362] Lustre: 25143:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 366.706421] Lustre: 25143:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 366.709286] Lustre: 25143:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 366.712386] Lustre: 25143:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 366.715879] Lustre: 25143:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 366.717892] Lustre: 25143:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 373.332590] Lustre: 25145:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 373.337384] Lustre: 25145:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 373.340196] Lustre: 25145:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 373.342154] Lustre: 25145:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 373.344453] Lustre: 25145:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 373.346308] Lustre: 25145:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 373.348463] Lustre: 25145:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 373.350970] Lustre: 25145:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 373.353969] Lustre: 25145:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 373.356733] Lustre: 25145:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 373.358676] Lustre: 25145:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 373.361308] Lustre: 25145:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 383.036058] Lustre: 25144:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 383.040765] Lustre: 25144:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 383.044645] Lustre: 25144:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 383.047518] Lustre: 25144:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 383.050332] Lustre: 25144:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 383.053598] Lustre: 25144:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 383.057237] Lustre: 25144:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 383.060602] Lustre: 25144:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 383.064108] Lustre: 25144:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 383.067536] Lustre: 25144:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 383.071049] Lustre: 25144:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 383.074443] Lustre: 25144:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 424.928767] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 424.929176] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 424.936995] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 430.048708] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 430.154527] Lustre: server umount lustre-OST07c6 complete [ 433.353608] Lustre: server umount lustre-MDT0000 complete [ 439.191434] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 440.259250] Key type lgssc unregistered [ 440.391450] LNet: 28733:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 440.394468] LNetError: 28733:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 440.404485] LNet: Removed LNI 192.168.204.136@tcp [ 440.704148] Key type .llcrypt unregistered [ 440.705763] Key type ._llcrypt unregistered [ 448.083968] Key type ._llcrypt registered [ 448.084945] Key type .llcrypt registered [ 448.116298] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 448.481866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 448.497943] alg: No test for adler32 (adler32-zlib) [ 449.346366] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 449.426677] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 451.007144] Key type lgssc registered [ 451.365185] Lustre: Echo OBD driver; http://www.lustre.org/ [ 454.020387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 454.026065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 455.183264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 456.625374] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 459.541210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 459.675082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 461.079453] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 463.529264] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 465.440891] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 467.982435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 468.151878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.214815] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 472.714638] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 480.422967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 489.951948] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 489.954535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.959831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 490.976750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 490.977059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 490.978014] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 494.096941] Lustre: server umount lustre-OST0000 complete [ 495.584027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 495.587071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.591318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 495.593423] Lustre: Skipped 1 previous similar message [ 497.494364] Lustre: server umount lustre-MDT0000 complete [ 498.766411] LustreError: 29961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781661336 with bad export cookie 7128209137376213427 [ 498.769703] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 498.771185] LustreError: 29961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 500.909663] Lustre: server umount lustre-MDT0001 complete [ 504.395964] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 505.474895] Key type lgssc unregistered [ 505.602452] LNet: 33732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 505.604739] LNetError: 33732:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 505.612107] LNet: Removed LNI 192.168.204.136@tcp [ 505.884997] Key type .llcrypt unregistered [ 505.886103] Key type ._llcrypt unregistered [ 512.957319] Key type ._llcrypt registered [ 512.958315] Key type .llcrypt registered [ 512.994431] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 513.268794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 513.277818] alg: No test for adler32 (adler32-zlib) [ 514.124946] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 514.208581] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 515.783168] Key type lgssc registered [ 516.097064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 518.396912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 518.402155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 519.540801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 520.790889] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 523.439981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 523.548513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 524.734256] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 526.957374] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 528.721575] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 530.954310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 531.085913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 532.925282] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 535.150964] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 545.447549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 550.881222] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 550.881526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 550.884133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 550.887229] Lustre: Skipped 1 previous similar message [ 550.891920] Lustre: Skipped 1 previous similar message [ 551.767202] Lustre: server umount lustre-OST0000 complete [ 554.463754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 554.466524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.470692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 555.160910] Lustre: server umount lustre-MDT0000 complete [ 556.000958] LustreError: 35706:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 556.001509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.007715] LustreError: 35706:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 556.411306] LustreError: 34960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781661393 with bad export cookie 3476419282531292193 [ 556.414089] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 556.415300] LustreError: 34960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 556.528600] Lustre: server umount lustre-MDT0001 complete [ 559.925060] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 560.892118] Key type lgssc unregistered [ 561.021397] LNet: 38530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 561.023812] LNetError: 38530:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 561.033429] LNet: Removed LNI 192.168.204.136@tcp [ 561.341496] Key type .llcrypt unregistered [ 561.342733] Key type ._llcrypt unregistered [ 570.302288] Key type ._llcrypt registered [ 570.303748] Key type .llcrypt registered [ 570.353087] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 570.694773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 570.704269] alg: No test for adler32 (adler32-zlib) [ 571.567383] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 571.663474] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 573.255190] Key type lgssc registered [ 573.677780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 576.456983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 576.464458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 577.676842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 579.141745] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 582.231424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 582.388181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 583.952038] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 586.611914] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 588.463078] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 590.573636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 590.695242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 592.443991] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 594.702384] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 601.958424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 608.223739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 608.226590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 608.231375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 610.415601] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.36@tcp (stopping) [ 612.321188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 612.321506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 612.323566] Lustre: Skipped 1 previous similar message [ 612.328217] Lustre: Skipped 1 previous similar message [ 613.785387] Lustre: server umount lustre-MDT0000 complete [ 615.021038] LustreError: 41762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781661452 with bad export cookie 2886516874965786057 [ 615.024126] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 615.025836] LustreError: 41762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 615.526767] LustreError: 41932:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 615.534507] LustreError: 41932:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 615.534886] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.36@tcp (stopping) [ 615.539471] Lustre: Skipped 1 previous similar message [ 617.439825] LustreError: 39947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 617.440111] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.447648] LustreError: 39919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 620.447905] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 620.456584] Lustre: Skipped 1 previous similar message [ 621.163583] Lustre: server umount lustre-MDT0001 complete [ 647.135107] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 647.190914] Lustre: server umount lustre-OST0000 complete [ 652.563726] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 653.603093] Key type lgssc unregistered [ 653.731377] LNet: 43802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.734873] LNetError: 43802:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 653.744523] LNet: Removed LNI 192.168.204.136@tcp [ 654.038850] Key type .llcrypt unregistered [ 654.040053] Key type ._llcrypt unregistered [ 662.573036] Key type ._llcrypt registered [ 662.573974] Key type .llcrypt registered [ 662.609698] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 662.905615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 662.943346] alg: No test for adler32 (adler32-zlib) [ 663.797442] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 663.871362] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 665.447170] Key type lgssc registered [ 665.797550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 668.195200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 668.200319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.342650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 670.543399] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 673.126141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 673.230836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 674.552405] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 676.836920] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 678.620311] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 680.672380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 680.800951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 682.600821] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 684.817362] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 694.117780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 698.335677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 698.338771] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 698.343032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 699.361037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 699.361992] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 699.362367] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 700.527359] Lustre: lustre-OST0000: Not available for connect from 192.168.204.36@tcp (stopping) [ 700.529090] Lustre: Skipped 1 previous similar message [ 703.631337] Lustre: server umount lustre-OST0000 complete [ 716.241558] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 716.243197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 716.245666] Lustre: Skipped 1 previous similar message [ 720.287980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 720.289878] Lustre: Skipped 2 previous similar messages [ 722.450190] Lustre: server umount lustre-MDT0000 complete [ 723.639413] LustreError: 45180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781661560 with bad export cookie 3044049126277238171 [ 723.642405] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 723.646806] LustreError: 45180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 729.833424] Lustre: server umount lustre-MDT0001 complete [ 733.188522] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 734.221839] Key type lgssc unregistered [ 734.352433] LNet: 48924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.354756] LNetError: 48924:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.361457] LNet: Removed LNI 192.168.204.136@tcp [ 734.655542] Key type .llcrypt unregistered [ 734.656592] Key type ._llcrypt unregistered [ 743.227595] Key type ._llcrypt registered [ 743.228525] Key type .llcrypt registered [ 743.260545] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 743.561931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 743.583357] alg: No test for adler32 (adler32-zlib) [ 744.431533] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 744.516119] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 746.095171] Key type lgssc registered [ 746.443316] Lustre: Echo OBD driver; http://www.lustre.org/ [ 748.923668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 748.930153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 750.086909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 751.434929] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 754.159448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 754.293343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 755.615547] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 757.798069] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 759.512353] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 761.571440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 761.673676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 763.429480] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 765.627757] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 776.037882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 778.237376] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 779.236251] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 789.847989] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-1622h00m26s [ 790.316293] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 791.523411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 791.524754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 791.527225] Lustre: Skipped 2 previous similar messages [ 791.528915] Lustre: Skipped 1 previous similar message [ 792.545372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 792.548293] Lustre: Skipped 3 previous similar messages [ 798.690636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 798.692433] Lustre: Skipped 3 previous similar messages [ 803.878423] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.36@tcp (stopping) [ 803.880711] Lustre: Skipped 1 previous similar message [ 805.343128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 805.401388] Lustre: server umount lustre-MDT0000 complete [ 806.429270] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 808.497450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 808.537122] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 808.622474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 808.649800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 809.636966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 809.934872] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 811.060440] Lustre: DEBUG MARKER: Wait for df (36402) ... [ 811.523791] Lustre: DEBUG MARKER: done [ 814.047938] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 814.050551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 814.054309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 814.056443] Lustre: Skipped 3 previous similar messages [ 814.473145] Lustre: server umount lustre-OST0000 complete [ 820.193133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 820.193797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 821.716191] Lustre: server umount lustre-MDT0000 complete [ 822.918988] LustreError: 50343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781661660 with bad export cookie 3368249750208446251 [ 822.921873] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 822.922471] LustreError: 50343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 833.541873] Lustre: server umount lustre-MDT0001 complete [ 836.769940] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 837.692271] Key type lgssc unregistered [ 837.814387] LNet: 55264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.817306] LNetError: 55264:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 837.824250] LNet: Removed LNI 192.168.204.136@tcp [ 838.091689] Key type .llcrypt unregistered [ 838.092699] Key type ._llcrypt unregistered [ 851.491449] Key type ._llcrypt registered [ 851.492732] Key type .llcrypt registered [ 851.530488] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 851.925559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 851.942518] alg: No test for adler32 (adler32-zlib) [ 852.796801] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 852.880622] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 854.455149] Key type lgssc registered [ 854.805438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 857.253927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 857.260612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.351529] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 858.363106] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 858.423168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 859.720720] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 862.457079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 862.490190] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 862.499948] Lustre: 57450:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 862.591474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 863.905112] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 866.174802] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 867.864166] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 870.074255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 870.143143] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 870.202191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 870.471709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 872.093625] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 874.480770] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 878.813440] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-1622h01m55s [ 879.348274] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 881.597453] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 882.380067] Lustre: DEBUG MARKER: Creating a test file and stat it [ 882.402396] LustreError: 59972:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 902.770301] Lustre: lustre-MDT0000: Client aa095f52-5e16-4cfd-a149-8046d1c2f50b (at 192.168.204.36@tcp) reconnecting [ 922.447106] LustreError: 59972:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 923.164078] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 923.988386] Lustre: DEBUG MARKER: done [ 926.693349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 926.693614] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 926.694719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 926.698297] Lustre: Skipped 1 previous similar message [ 926.700416] LustreError: Skipped 1 previous similar message [ 931.597308] Lustre: server umount lustre-OST0000 complete [ 934.367856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 934.370589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 934.375633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 934.378019] Lustre: Skipped 1 previous similar message [ 935.392277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 935.394168] Lustre: Skipped 2 previous similar messages [ 937.440519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 937.443330] Lustre: Skipped 1 previous similar message [ 943.585375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 943.587881] Lustre: Skipped 1 previous similar message [ 947.167091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 947.235099] Lustre: server umount lustre-MDT0000 complete [ 948.418306] LustreError: 57440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781661785 with bad export cookie 6396727547449820159 [ 948.420718] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 948.421838] LustreError: 57440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 948.540730] Lustre: server umount lustre-MDT0001 complete [ 952.042265] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 953.101743] Key type lgssc unregistered [ 953.241404] LNet: 61773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.243530] LNetError: 61773:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.252281] LNet: Removed LNI 192.168.204.136@tcp [ 953.532889] Key type .llcrypt unregistered [ 953.533884] Key type ._llcrypt unregistered [ 968.768980] Key type ._llcrypt registered [ 968.770668] Key type .llcrypt registered [ 968.810926] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 969.102155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 969.112185] alg: No test for adler32 (adler32-zlib) [ 969.967921] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 970.049991] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 971.631184] Key type lgssc registered [ 971.984885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 973.067311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 973.073939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 974.174137] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 974.182506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 974.239895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 974.577556] Lustre: Failing over lustre-MDT0000 [ 974.683321] Lustre: server umount lustre-MDT0000 complete [ 976.738730] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 22:03:33 (1781661813) [ 979.015128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 979.170316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 980.539734] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 983.360306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 983.393714] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 983.406634] Lustre: 64548:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 983.503182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 984.841923] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 987.256132] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 988.873892] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 990.858759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 990.923221] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 990.976274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 992.645827] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 992.976222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 994.832303] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 997.793306] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1003.488722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1003.490272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1003.493102] Lustre: Skipped 1 previous similar message [ 1005.594940] Lustre: server umount lustre-MDT0000 complete [ 1006.823163] LustreError: 66399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781661844 with bad export cookie 8348308589997236275 [ 1006.826373] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1006.828956] LustreError: 66399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1006.932839] Lustre: server umount lustre-MDT0001 complete [ 1008.055928] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1009.695193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1012.543855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1016.735717] LustreError: 63468:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000003b5c4a x1868207810967680/t0(0) o250->MGC192.168.204.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1016.884941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1016.917038] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1018.223573] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1020.960139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1021.059123] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1021.064786] Lustre: Skipped 1 previous similar message [ 1022.506288] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1023.142493] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1024.159089] Lustre: 63470:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781661845/real 1781661845] req@000000004a8e541a x1868207810967296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781661861 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1024.911233] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1026.505047] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1027.215080] Lustre: 63472:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781661848/real 1781661848] req@00000000ae1e973c x1868207810967424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781661864 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1028.576389] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1028.577782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1028.579473] Lustre: Skipped 1 previous similar message [ 1028.582408] Lustre: Skipped 4 previous similar messages [ 1029.599149] Lustre: 63472:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781661850/real 1781661850] req@0000000002fd1076 x1868207810967552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781661866 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1033.695152] Lustre: 63469:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781661854/real 1781661854] req@000000005c10919b x1868207810967808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781661870 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1033.696934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1033.703936] Lustre: Skipped 2 previous similar messages [ 1034.197969] Lustre: server umount lustre-MDT0000 complete [ 1035.333029] LustreError: 64539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781661872 with bad export cookie 8348308589997238543 [ 1035.336217] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1035.336562] LustreError: 64539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1035.432258] Lustre: server umount lustre-MDT0001 complete [ 1037.383346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1040.283429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1044.959559] LustreError: 63468:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000ffea154 x1868207810986880/t0(0) o250->MGC192.168.204.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1045.059123] LustreError: 68332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1045.059718] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1045.064050] LustreError: 68332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1045.071584] Lustre: Skipped 1 previous similar message [ 1045.083761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1045.085652] Lustre: Skipped 1 previous similar message [ 1045.104907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1046.329221] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1048.932759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1050.255439] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1052.451421] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1054.143051] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1054.180515] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1054.183596] Lustre: Skipped 1 previous similar message [ 1055.199164] Lustre: 63471:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781661876/real 1781661876] req@00000000c19ea18c x1868207810986496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781661892 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1058.784958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1058.786044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1058.787493] Lustre: Skipped 1 previous similar message [ 1058.794151] Lustre: Skipped 1 previous similar message [ 1061.839780] Lustre: server umount lustre-MDT0000 complete [ 1063.050419] LustreError: 65283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781661900 with bad export cookie 8348308589997239922 [ 1063.054799] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1063.055049] LustreError: 65283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1063.153388] Lustre: server umount lustre-MDT0001 complete [ 1065.172818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1066.194750] Lustre: DEBUG MARKER: files compared the same [ 1077.089128] Lustre: server umount lustre-OST0000 complete [ 1082.115377] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1083.213719] Key type lgssc unregistered [ 1083.352420] LNet: 74261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.354899] LNetError: 74261:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1083.363405] LNet: Removed LNI 192.168.204.136@tcp [ 1083.688097] Key type .llcrypt unregistered [ 1083.689248] Key type ._llcrypt unregistered [ 1092.541461] Key type ._llcrypt registered [ 1092.542450] Key type .llcrypt registered [ 1092.579823] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1092.933628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1092.955243] alg: No test for adler32 (adler32-zlib) [ 1093.801463] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 1093.878557] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1095.463167] Key type lgssc registered [ 1095.811728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1098.454551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1098.462534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1099.624072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1100.972796] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug +malloc all [ 1103.762635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.858086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1105.198434] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug +malloc all [ 1107.590540] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1109.447439] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1111.723775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1111.822937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1113.743253] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug +malloc all [ 1116.153579] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1124.131931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1129.440635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1129.440758] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1129.444552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1129.448184] Lustre: Skipped 1 previous similar message [ 1131.016707] Lustre: server umount lustre-OST0000 complete [ 1134.289428] Lustre: server umount lustre-MDT0000 complete [ 1134.559668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1134.559744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.562297] LustreError: Skipped 1 previous similar message [ 1134.567217] Lustre: Skipped 1 previous similar message [ 1134.568838] LustreError: 76392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1134.574660] LustreError: 76392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1135.489538] LustreError: 75633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781661972 with bad export cookie 16304135766474112245 [ 1135.490924] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1135.493454] LustreError: 75633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1135.594739] Lustre: server umount lustre-MDT0001 complete [ 1139.210736] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1140.269331] Key type lgssc unregistered [ 1140.398323] LNet: 79210:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1140.400567] LNetError: 79210:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1140.408231] LNet: Removed LNI 192.168.204.136@tcp [ 1140.660755] Key type .llcrypt unregistered [ 1140.661647] Key type ._llcrypt unregistered [ 1149.205127] Key type ._llcrypt registered [ 1149.206833] Key type .llcrypt registered [ 1149.243452] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1149.577275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1149.606656] alg: No test for adler32 (adler32-zlib) [ 1150.459862] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 1150.542305] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1152.127162] Key type lgssc registered [ 1152.505716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1155.103185] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1155.110639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1170.527298] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1175.647258] LustreError: 80577:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.136@tcp: failed processing log, type 4: rc = -110 [ 1202.271135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1204.173122] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1206.569209] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1209.033658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1209.076156] Lustre: *** cfs_fail_loc=706, val=0*** [ 1209.220237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1209.246745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1210.526320] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1213.089544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1213.209719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1214.412600] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1216.686377] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1218.323348] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1219.551835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1219.554816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1219.560111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1222.624564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1222.626908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1227.746781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1227.749340] Lustre: Skipped 2 previous similar messages [ 1232.865582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1232.867439] Lustre: Skipped 1 previous similar message [ 1233.375095] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1233.423899] Lustre: server umount lustre-OST0000 complete [ 1237.984627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.985399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1237.988090] Lustre: Skipped 1 previous similar message [ 1240.725563] Lustre: server umount lustre-MDT0000 complete [ 1241.877129] LustreError: 80584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662079 with bad export cookie 13681310107481451353 [ 1241.880110] LustreError: 80584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1241.880376] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1241.983665] Lustre: server umount lustre-MDT0001 complete [ 1245.297062] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1246.301511] Key type lgssc unregistered [ 1246.430354] LNet: 84188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1246.433170] LNetError: 84188:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1246.442277] LNet: Removed LNI 192.168.204.136@tcp [ 1246.705252] Key type .llcrypt unregistered [ 1246.706129] Key type ._llcrypt unregistered [ 1255.339103] Key type ._llcrypt registered [ 1255.339939] Key type .llcrypt registered [ 1255.374530] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1255.761526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1255.778247] alg: No test for adler32 (adler32-zlib) [ 1256.634475] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 1256.715864] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1258.295099] Key type lgssc registered [ 1258.651867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1261.218744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1261.224322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1263.514384] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1265.900666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1266.077555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1267.386070] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1270.113780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1270.224162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1272.060586] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1274.684696] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1274.689784] Lustre: MGS: Client 333537ed-5361-4ba5-a529-93f6065b565c (at 0@lo) reconnecting [ 1274.781170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1274.814775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1276.076237] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1285.087981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1285.089803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1285.092967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1286.115421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1286.115824] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1286.116921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.940064] Lustre: server umount lustre-OST0000 complete [ 1291.232336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1291.232466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1291.239875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1291.241513] Lustre: Skipped 2 previous similar messages [ 1296.352154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1296.354052] Lustre: Skipped 1 previous similar message [ 1297.219921] Lustre: server umount lustre-MDT0000 complete [ 1298.555306] Lustre: server umount lustre-MDT0001 complete [ 1299.831907] Lustre: server umount MGS complete [ 1303.575834] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 22:08:59 (1781662139) [ 1312.348932] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 1315.061786] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1325.591582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1326.877962] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1330.534403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1330.619695] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1330.635790] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1330.671828] Lustre: lustre-MDT0001: new disk, initializing [ 1330.695608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1330.702974] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1330.705679] Lustre: 93427:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1330.708895] LustreError: 93427:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 1, retries 0, failed: rc = -115 [ 1331.932764] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1335.940634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1336.034818] Lustre: lustre-OST0000: new disk, initializing [ 1336.036464] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1336.038621] Lustre: 94192:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1336.061525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1337.988420] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1340.959763] Lustre: 94709:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000001175f8a9 x1868208111911680/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781662194 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1341.473464] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1343.489454] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1343.492965] Lustre: MGS: Client 62ab1bb0-6faa-4f59-bbaf-95b868b39091 (at 0@lo) reconnecting [ 1343.503322] Lustre: 95237:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1343.508548] LustreError: 93411:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1343.510841] LustreError: 93411:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1343.515565] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1343.545726] Lustre: lustre-MDT0000: new disk, initializing [ 1343.593569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1344.607782] Lustre: 95264:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000a826c0c5 x1868208111917056/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1781662197 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1344.671638] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1344.841261] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1346.657906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1346.660604] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1346.728169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1363.936190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1363.938988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1363.942456] Lustre: Skipped 1 previous similar message [ 1363.944084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1369.420837] Lustre: server umount lustre-OST0000 complete [ 1372.128981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1372.128996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1372.129761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1372.129765] Lustre: Skipped 3 previous similar messages [ 1372.133859] Lustre: Skipped 2 previous similar messages [ 1372.135758] LustreError: Skipped 1 previous similar message [ 1376.833834] Lustre: server umount lustre-MDT0000 complete [ 1377.248526] LustreError: 95251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1377.255520] LustreError: 95251:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 1384.180611] Lustre: server umount lustre-MDT0001 complete [ 1385.377750] Lustre: server umount MGS complete [ 1389.199118] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 22:10:25 (1781662225) [ 1395.103220] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1396.109656] Key type lgssc unregistered [ 1396.241368] LNet: 97756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1396.245539] LNetError: 97756:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1396.253360] LNet: Removed LNI 192.168.204.136@tcp [ 1396.534201] Key type .llcrypt unregistered [ 1396.535410] Key type ._llcrypt unregistered [ 1403.303299] Key type ._llcrypt registered [ 1403.304230] Key type .llcrypt registered [ 1403.342722] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1403.651170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1403.695559] alg: No test for adler32 (adler32-zlib) [ 1404.558896] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 1404.638612] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1406.215186] Key type lgssc registered [ 1406.587836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1409.019536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1409.025450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1410.191199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1411.436913] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1413.987682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1414.091305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1415.304738] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1417.428629] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1419.053994] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1421.119054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1421.223323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1423.048951] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1425.294604] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1435.558693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1439.712206] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1439.715099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1439.720195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1440.736749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1440.738146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1440.738382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1440.748107] Lustre: Skipped 1 previous similar message [ 1444.297345] Lustre: server umount lustre-OST0000 complete [ 1445.856345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.857086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1445.859798] Lustre: Skipped 1 previous similar message [ 1450.976129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1450.978568] Lustre: Skipped 2 previous similar messages [ 1451.666164] Lustre: server umount lustre-MDT0000 complete [ 1452.825106] LustreError: 98936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662290 with bad export cookie 16745836732210851054 [ 1452.828260] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1452.828499] LustreError: 98936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1452.933345] Lustre: server umount lustre-MDT0001 complete [ 1456.373173] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1457.225255] Key type lgssc unregistered [ 1457.344305] LNet: 102503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1457.346992] LNetError: 102503:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1457.353232] LNet: Removed LNI 192.168.204.136@tcp [ 1457.610888] Key type .llcrypt unregistered [ 1457.612086] Key type ._llcrypt unregistered [ 1463.824187] Key type ._llcrypt registered [ 1463.825710] Key type .llcrypt registered [ 1463.870935] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1464.259090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1464.326089] alg: No test for adler32 (adler32-zlib) [ 1465.191376] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 1465.279125] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1466.863118] Key type lgssc registered [ 1467.214815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1469.327622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1469.328679] LustreError: 103564:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1469.330536] Lustre: Skipped 1 previous similar message [ 1469.334182] LustreError: 103563:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1469.335875] LustreError: 103564:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4998 [ 1469.338296] LustreError: 103564:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1469.341311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1469.341703] LustreError: 103564:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1470.502246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1472.915531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1473.019830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1474.354725] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1475.862219] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing lsmod [ 1476.835717] LustreError: 104762:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1476.846824] LustreError: 104767:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1476.850332] LustreError: 104762:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4989 [ 1476.853253] LustreError: 104762:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1476.856690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1476.858288] LustreError: 104762:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1476.980337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1477.874612] LustreError: 103629:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1477.877201] LustreError: 103629:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1477.879370] LustreError: 103629:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 1478.111822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1478.113934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1478.117467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1482.208455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1482.209868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1482.210903] Lustre: Skipped 1 previous similar message [ 1482.214345] Lustre: Skipped 1 previous similar message [ 1483.233822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1483.235756] Lustre: Skipped 2 previous similar messages [ 1484.061228] Lustre: server umount lustre-MDT0000 complete [ 1485.314207] LustreError: 103577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662322 with bad export cookie 11857302361724377520 [ 1485.317882] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1485.319649] LustreError: 103577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1487.327327] LustreError: 104029:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000cd121904 x1868208329996928/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1487.378851] Lustre: server umount lustre-MDT0001 complete [ 1497.811839] Lustre: server umount lustre-OST0000 complete [ 1500.330901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1500.474486] LustreError: 105830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1500.479920] LustreError: 105830:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1500.496312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1501.721728] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1504.539670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1504.646246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1505.912437] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1508.145677] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1509.851491] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1512.126821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1514.102723] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1516.286953] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1525.542842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1529.823765] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1529.827030] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1529.832658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1530.849778] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1534.286750] Lustre: server umount lustre-OST0000 complete [ 1535.455667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1535.457949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1535.460847] Lustre: Skipped 1 previous similar message [ 1535.462497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1535.464009] Lustre: Skipped 2 previous similar messages [ 1541.525828] Lustre: server umount lustre-MDT0000 complete [ 1542.765489] LustreError: 105812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662380 with bad export cookie 11857302361724378171 [ 1542.768764] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1542.769946] LustreError: 105812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1542.897353] Lustre: server umount lustre-MDT0001 complete [ 1546.154073] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1547.092465] Key type lgssc unregistered [ 1547.215339] LNet: 109380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1547.218987] LNetError: 109380:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1547.226306] LNet: Removed LNI 192.168.204.136@tcp [ 1547.495909] Key type .llcrypt unregistered [ 1547.496849] Key type ._llcrypt unregistered [ 1555.780748] Key type ._llcrypt registered [ 1555.781745] Key type .llcrypt registered [ 1555.817864] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1556.210394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1556.223755] alg: No test for adler32 (adler32-zlib) [ 1557.073916] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 1557.150350] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1558.727089] Key type lgssc registered [ 1559.052413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1561.455247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1561.460914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1562.608594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1563.854899] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1566.472735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1566.574563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1567.742201] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1569.815652] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1571.359155] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1573.562435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1573.668590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1575.545363] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1577.937884] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1588.006054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1590.283548] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1591.040419] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1593.314038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1593.314191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1593.315268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1593.322125] Lustre: Skipped 1 previous similar message [ 1593.998668] Lustre: server umount lustre-OST0000 complete [ 1597.393355] Lustre: server umount lustre-MDT0000 complete [ 1597.408682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1597.410827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1597.414296] LustreError: 110773:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1597.418553] LustreError: 110773:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1598.432284] LustreError: 110791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1598.432458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1598.482926] LustreError: 110754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662435 with bad export cookie 10981300870661037344 [ 1598.486364] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1598.487019] LustreError: 110754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1598.583282] Lustre: server umount lustre-MDT0001 complete [ 1601.601183] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1602.556782] Key type lgssc unregistered [ 1602.695542] LNet: 114424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1602.698068] LNetError: 114424:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1602.705425] LNet: Removed LNI 192.168.204.136@tcp [ 1603.011809] Key type .llcrypt unregistered [ 1603.013656] Key type ._llcrypt unregistered [ 1609.983079] Key type ._llcrypt registered [ 1609.984034] Key type .llcrypt registered [ 1610.023719] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1610.474523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1610.485697] alg: No test for adler32 (adler32-zlib) [ 1611.337868] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 1611.423770] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1613.007158] Key type lgssc registered [ 1613.376635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1615.905339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1615.913029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1617.068029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1618.352749] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1621.169491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1621.289106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1622.668847] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1624.989587] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1626.710844] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1628.897139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1629.011829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1630.901710] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1633.240654] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1642.343975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 1646.559855] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1646.565390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1646.571627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1647.585651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1647.585914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1647.586211] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1648.280988] Lustre: server umount lustre-OST0000 complete [ 1651.669821] Lustre: server umount lustre-MDT0000 complete [ 1652.191896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1652.194250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1652.200395] LustreError: 115622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1652.205274] LustreError: 115622:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1652.705313] LustreError: 117460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1652.873122] LustreError: 115603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662490 with bad export cookie 6122622982373497140 [ 1652.876433] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1652.876431] LustreError: 115603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1652.986418] Lustre: server umount lustre-MDT0001 complete [ 1656.523276] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1657.571519] Key type lgssc unregistered [ 1657.705382] LNet: 119172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1657.708955] LNetError: 119172:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1657.718468] LNet: Removed LNI 192.168.204.136@tcp [ 1658.028260] Key type .llcrypt unregistered [ 1658.029679] Key type ._llcrypt unregistered [ 1667.025136] Key type ._llcrypt registered [ 1667.026174] Key type .llcrypt registered [ 1667.069648] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1667.565046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1667.580625] alg: No test for adler32 (adler32-zlib) [ 1668.432896] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 1668.523389] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1670.103160] Key type lgssc registered [ 1670.466967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1672.940395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1672.946688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1674.102829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1675.487401] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1678.231452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1678.364083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1679.760311] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1682.094242] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1683.803546] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1685.956260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1686.066832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1687.957165] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1690.262458] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1699.366971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 1703.398876] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1704.628065] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1704.630235] Lustre: Skipped 2 previous similar messages [ 1705.984450] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1705.986677] Lustre: Skipped 2 previous similar messages [ 1712.636104] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1712.645964] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1715.830551] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.36@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1715.833678] Lustre: Skipped 2 previous similar messages [ 1718.392820] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.36@tcp 192.168.0.[2,10]@tcp [ 1721.803185] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.36/24@tcp in log lustre-MDT0000 [ 1721.806103] Lustre: Skipped 2 previous similar messages [ 1724.085528] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.36/24@tcp [ 1724.087297] Lustre: Skipped 1 previous similar message [ 1728.991931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1728.995168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1728.999289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1730.016938] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1730.017646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1730.019819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1730.029200] Lustre: Skipped 1 previous similar message [ 1734.988459] Lustre: server umount lustre-OST0000 complete [ 1739.743739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1739.745662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1739.749047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1742.417714] Lustre: server umount lustre-MDT0000 complete [ 1743.511123] LustreError: 120546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662580 with bad export cookie 8199714781117871372 [ 1743.514257] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1743.515439] LustreError: 120546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1743.619662] Lustre: server umount lustre-MDT0001 complete [ 1746.855207] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1747.814674] Key type lgssc unregistered [ 1747.929384] LNet: 125564:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1747.931501] LNetError: 125564:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1747.940248] LNet: Removed LNI 192.168.204.136@tcp [ 1748.215480] Key type .llcrypt unregistered [ 1748.216403] Key type ._llcrypt unregistered [ 1757.549336] Key type ._llcrypt registered [ 1757.550149] Key type .llcrypt registered [ 1757.583119] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1757.899307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1757.936518] alg: No test for adler32 (adler32-zlib) [ 1758.770289] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 1758.846166] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1760.423156] Key type lgssc registered [ 1760.772552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1763.389812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1763.398932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1764.546738] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1764.552336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1764.555682] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.36/24@tcp [ 1765.775223] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1768.490888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1768.607432] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1768.615398] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.36/24@tcp [ 1768.634297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1769.971692] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1772.312555] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1774.098770] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1776.283249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1776.396311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1778.248794] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1780.600894] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1789.736097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 1795.041457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1795.041794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1795.043439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1795.048563] Lustre: Skipped 1 previous similar message [ 1796.172127] Lustre: server umount lustre-OST0000 complete [ 1799.377810] Lustre: server umount lustre-MDT0000 complete [ 1799.647792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1799.649825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1799.653562] LustreError: 127100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1799.660229] LustreError: 127100:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1800.160277] LustreError: 128537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1800.164276] LustreError: 128537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1800.467065] LustreError: 127083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662637 with bad export cookie 8916016123018889280 [ 1800.469914] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1800.471574] LustreError: 127083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1800.579590] Lustre: server umount lustre-MDT0001 complete [ 1803.707486] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1804.662989] Key type lgssc unregistered [ 1804.789715] LNet: 130705:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1804.792603] LNetError: 130705:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1804.804406] LNet: Removed LNI 192.168.204.136@tcp [ 1805.088746] Key type .llcrypt unregistered [ 1805.089994] Key type ._llcrypt unregistered [ 1813.347334] Key type ._llcrypt registered [ 1813.348516] Key type .llcrypt registered [ 1813.381960] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 1819.343424] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1819.726452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1819.756075] alg: No test for adler32 (adler32-zlib) [ 1820.610655] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 1820.695618] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1822.279134] Key type lgssc registered [ 1822.638976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1833.055645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1833.063621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1834.144797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1834.157530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1834.190778] Lustre: lustre-MDT0000: new disk, initializing [ 1834.209917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1834.215123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1835.386223] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1839.287461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1839.334228] Lustre: 134077:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1839.363401] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1839.365083] Lustre: Skipped 1 previous similar message [ 1839.400894] Lustre: lustre-MDT0001: new disk, initializing [ 1839.435043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1839.449357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1839.455830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1840.762225] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1844.418942] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1846.132597] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1848.367618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1848.479941] Lustre: lustre-OST0000: new disk, initializing [ 1848.481838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1848.484371] Lustre: 136041:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1848.518962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1850.405561] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1854.037111] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1855.483765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1855.487628] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1855.524357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1855.548203] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1855.627414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1856.956615] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1857.031244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1860.577883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1860.578022] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1860.579405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1860.580477] LustreError: Skipped 1 previous similar message [ 1860.585202] Lustre: Skipped 1 previous similar message [ 1863.756733] Lustre: server umount lustre-OST0000 complete [ 1865.184125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1865.188680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1865.196293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1865.198582] Lustre: Skipped 1 previous similar message [ 1870.816391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1870.818070] Lustre: Skipped 2 previous similar messages [ 1871.126353] Lustre: server umount lustre-MDT0000 complete [ 1872.345362] LustreError: 134068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662709 with bad export cookie 8934669694790488582 [ 1872.348588] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1872.349619] LustreError: 134068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1872.464780] Lustre: server umount lustre-MDT0001 complete [ 1875.267542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1875.443417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1876.766254] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1879.527071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1879.646437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1881.055859] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1883.426056] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1885.198886] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1887.569557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1887.682634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1889.590536] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1891.938540] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1893.344041] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1900.555454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 1901.803712] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1901.872715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1905.025555] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1905.085338] Lustre: lustre-OST0001: new disk, initializing [ 1905.087256] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1905.089254] Lustre: 141677:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1905.115088] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1906.973372] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1910.344199] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1911.809501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1911.812799] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1911.843264] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1912.462818] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 1913.860873] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1913.928369] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1915.579146] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c03c4bbb800.ost_server_uuid 50 [ 1916.039671] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8c03c4bbb800.ost_server_uuid in FULL state after 0 sec [ 1922.017935] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1922.020299] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1922.023599] Lustre: Skipped 1 previous similar message [ 1922.025401] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1922.027011] Lustre: Skipped 1 previous similar message [ 1923.838558] Lustre: server umount lustre-OST0001 complete [ 1925.600185] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1925.602024] LustreError: Skipped 1 previous similar message [ 1925.603054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1925.605913] Lustre: Skipped 1 previous similar message [ 1927.136968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1927.138452] LustreError: 140074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1927.138809] Lustre: Skipped 1 previous similar message [ 1927.143083] LustreError: 140074:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1931.081244] Lustre: server umount lustre-OST0000 complete [ 1932.256452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1932.261154] Lustre: Skipped 1 previous similar message [ 1937.376069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1937.377910] Lustre: Skipped 3 previous similar messages [ 1938.389578] Lustre: server umount lustre-MDT0000 complete [ 1939.598312] LustreError: 138234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662776 with bad export cookie 8934669694790489450 [ 1939.600880] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1939.603795] LustreError: 138234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1939.714643] Lustre: server umount lustre-MDT0001 complete [ 1945.675208] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1946.647871] Key type lgssc unregistered [ 1946.769422] LNet: 144687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1946.771731] LNetError: 144687:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1946.782420] LNet: Removed LNI 192.168.204.136@tcp [ 1947.069134] Key type .llcrypt unregistered [ 1947.070598] Key type ._llcrypt unregistered [ 1958.906627] Key type ._llcrypt registered [ 1958.907981] Key type .llcrypt registered [ 1958.941146] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 1964.442451] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1964.850627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1964.868056] alg: No test for adler32 (adler32-zlib) [ 1965.709367] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 1965.795791] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1967.375162] Key type lgssc registered [ 1967.722484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1978.657441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1978.665218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1979.757109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1979.769499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1979.806472] Lustre: lustre-MDT0000: new disk, initializing [ 1979.831922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1979.839951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1981.204181] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1985.361768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1985.405664] Lustre: 148812:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1985.431716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1985.434932] Lustre: Skipped 1 previous similar message [ 1985.470198] Lustre: lustre-MDT0001: new disk, initializing [ 1985.502776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1985.515941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1985.519267] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1986.877637] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1990.505378] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1992.273968] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1994.520080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1994.626634] Lustre: lustre-OST0000: new disk, initializing [ 1994.629664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1994.634475] Lustre: 150780:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1994.670408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1996.612572] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2000.405568] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2003.962331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2003.969717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2004.003459] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2007.407656] Lustre: Failing over lustre-OST0000 [ 2007.474479] Lustre: server umount lustre-OST0000 complete [ 2020.265026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2020.386087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2020.389685] Lustre: Skipped 1 previous similar message [ 2020.396545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2020.404722] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2022.000054] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2022.310139] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2023.557303] Lustre: Failing over lustre-MDT0000 [ 2023.656331] Lustre: server umount lustre-MDT0000 complete [ 2025.441589] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2025.446016] Lustre: Skipped 1 previous similar message [ 2025.615090] Lustre: 147040:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781662846/real 1781662846] req@000000007ca7437c x1868208855345792/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781662862 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2026.463107] Lustre: 147039:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781662846/real 1781662846] req@000000005e653344 x1868208855346560/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781662862 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2030.559155] Lustre: 147039:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781662851/real 1781662851] req@00000000bf8fd1c7 x1868208855347712/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781662867 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2030.560603] LustreError: 149623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2030.578814] LustreError: 149623:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2032.230946] LustreError: 151500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2035.679083] Lustre: 147042:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781662856/real 1781662856] req@0000000045412b03 x1868208855348864/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781662872 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2035.680696] LustreError: 148824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2035.694154] LustreError: 148824:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2036.552505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2036.594509] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2036.598813] Lustre: 148816:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.136@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2036.669227] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2036.686740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2036.717884] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2037.351078] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2037.971957] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2041.829072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2041.835985] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2041.839769] Lustre: Skipped 1 previous similar message [ 2041.856971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2046.946328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2046.949962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2046.952087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2046.954416] Lustre: Skipped 2 previous similar messages [ 2048.977677] Lustre: server umount lustre-OST0000 complete [ 2052.064803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2052.064950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2052.069127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2052.071222] LustreError: Skipped 1 previous similar message [ 2052.073701] Lustre: Skipped 2 previous similar messages [ 2056.341944] Lustre: server umount lustre-MDT0000 complete [ 2057.184072] LustreError: 149623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2057.188282] LustreError: 149623:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2057.593157] LustreError: 148804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781662894 with bad export cookie 2489539611761219755 [ 2057.596101] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2057.598414] LustreError: 148804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2057.725280] Lustre: server umount lustre-MDT0001 complete [ 2061.415780] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 2062.397521] Key type lgssc unregistered [ 2062.526319] LNet: 154426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2062.528727] LNetError: 154426:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2062.538348] LNet: Removed LNI 192.168.204.136@tcp [ 2062.826501] Key type .llcrypt unregistered [ 2062.827535] Key type ._llcrypt unregistered [ 2082.483725] Key type ._llcrypt registered [ 2082.485572] Key type .llcrypt registered [ 2082.523486] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2082.811299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2082.830643] alg: No test for adler32 (adler32-zlib) [ 2083.678327] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 2083.762508] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 2085.343124] Key type lgssc registered [ 2085.714798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2088.203568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2088.209044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2089.353431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2090.681186] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2094.234936] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2096.152256] LustreError: 155970:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2096.155220] LustreError: 155970:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2096.157699] LustreError: 155970:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2096.273433] Lustre: server umount lustre-MDT0000 complete [ 2101.051413] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 2104.151277] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2115.370685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.457883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2115.468475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2115.506893] Lustre: lustre-MDT0000: new disk, initializing [ 2115.530941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.537605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2116.910830] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2121.015904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2121.055465] Lustre: 159268:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2121.078553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2121.080355] Lustre: Skipped 1 previous similar message [ 2121.116550] Lustre: lustre-MDT0001: new disk, initializing [ 2121.148577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2121.160525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2121.164429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2122.439098] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2126.019643] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2127.757763] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2129.955090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2130.052392] Lustre: lustre-OST0000: new disk, initializing [ 2130.053988] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2130.056492] Lustre: 161236:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2130.084230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2131.944434] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2132.792824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2132.795573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2132.826051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2135.596972] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2137.443989] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2137.995853] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 22:22:54 (1781662974) [ 2199.519479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.519729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2199.523858] Lustre: Skipped 2 previous similar messages [ 2199.527150] Lustre: Skipped 2 previous similar messages [ 2204.639541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2204.641488] Lustre: Skipped 2 previous similar messages [ 2205.343714] Lustre: server umount lustre-MDT0000 complete [ 2206.547574] LustreError: 159259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663043 with bad export cookie 6784315680742098419 [ 2206.548595] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2206.552768] LustreError: 159259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2209.759394] LustreError: 162986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2209.759474] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2209.767122] LustreError: 162986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2209.771634] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2211.807490] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2211.807539] LustreError: 161955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2212.707715] Lustre: server umount lustre-MDT0001 complete [ 2220.107231] Lustre: server umount lustre-OST0000 complete [ 2223.002766] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 2226.129580] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2237.970536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2238.062668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2238.074867] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2238.116265] Lustre: lustre-MDT0000: new disk, initializing [ 2238.141168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2238.148411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2239.467135] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2243.538738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2243.581060] Lustre: 165860:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2243.605715] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2243.607408] Lustre: Skipped 1 previous similar message [ 2243.643181] Lustre: lustre-MDT0001: new disk, initializing [ 2243.685683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2243.688549] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2245.057301] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2248.661399] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2250.366596] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2252.537064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2252.634196] Lustre: lustre-OST0000: new disk, initializing [ 2252.636052] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2252.638485] Lustre: 167828:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2252.669347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2252.671589] Lustre: Skipped 1 previous similar message [ 2254.527128] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2255.287111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2255.291791] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2255.323737] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2258.169930] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2259.539977] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2259.609521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2261.080640] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2261.161089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2263.888289] Lustre: server umount lustre-OST0000 complete [ 2265.568561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2265.569015] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2265.573604] Lustre: Skipped 1 previous similar message [ 2275.809210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2275.811049] Lustre: Skipped 4 previous similar messages [ 2279.391127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2279.464304] Lustre: server umount lustre-MDT0000 complete [ 2280.679179] LustreError: 165854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663117 with bad export cookie 6784315680742257235 [ 2280.681590] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2280.682257] LustreError: 165854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2290.191626] Lustre: server umount lustre-MDT0001 complete [ 2292.852248] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:25:29 (1781663129) [ 2298.013282] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 2301.147448] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2312.917651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2313.031033] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2313.043633] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2313.095782] Lustre: lustre-MDT0000: new disk, initializing [ 2313.122665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2313.129349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2314.485594] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2318.658358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2318.703123] Lustre: 172585:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2318.804867] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2320.198586] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2323.944342] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2325.738331] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2327.929679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2328.029187] Lustre: 174564:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2328.496733] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2328.531719] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2329.990393] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2333.608541] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2338.784987] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2338.785402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2338.787734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2338.790806] Lustre: Skipped 1 previous similar message [ 2341.581563] Lustre: server umount lustre-OST0000 complete [ 2343.904909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2343.907932] Lustre: Skipped 1 previous similar message [ 2357.215082] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2357.279858] Lustre: server umount lustre-MDT0000 complete [ 2358.465211] LustreError: 173384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663195 with bad export cookie 6784315680742258103 [ 2358.468044] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2358.468230] LustreError: 173384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2358.580016] Lustre: server umount lustre-MDT0001 complete [ 2361.149714] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:26:37 (1781663197) [ 2366.095529] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 2369.039207] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2380.315447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2380.399194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2380.412159] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2380.414982] Lustre: Skipped 3 previous similar messages [ 2380.459893] Lustre: lustre-MDT0000: new disk, initializing [ 2380.461973] Lustre: Skipped 2 previous similar messages [ 2380.485980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2380.488077] Lustre: Skipped 2 previous similar messages [ 2380.493720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2380.496178] Lustre: Skipped 2 previous similar messages [ 2381.806673] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2385.808132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2385.847209] Lustre: 178877:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2385.881280] LustreError: 178886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2385.967120] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2387.217916] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2390.741212] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2392.447071] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2394.599522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2394.703321] Lustre: 180851:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2396.560191] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2400.077670] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2402.327621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2407.395219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2407.395540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2407.399103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2407.400619] LustreError: Skipped 2 previous similar messages [ 2407.402059] Lustre: Skipped 8 previous similar messages [ 2409.165847] Lustre: server umount lustre-OST0000 complete [ 2411.487804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2412.502599] Lustre: server umount lustre-MDT0000 complete [ 2412.513949] LustreError: 178886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2412.519723] LustreError: 178886:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2413.722099] LustreError: 181957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663251 with bad export cookie 6784315680742259174 [ 2413.726491] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2413.727402] LustreError: 181957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2413.830652] Lustre: server umount lustre-MDT0001 complete [ 2417.289310] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 2418.253504] Key type lgssc unregistered [ 2418.378368] LNet: 182706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2418.381237] LNetError: 182706:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2418.389914] LNet: Removed LNI 192.168.204.136@tcp [ 2418.687362] Key type .llcrypt unregistered [ 2418.688358] Key type ._llcrypt unregistered [ 2427.314912] Key type ._llcrypt registered [ 2427.316194] Key type .llcrypt registered [ 2427.356080] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2427.770780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2427.781847] alg: No test for adler32 (adler32-zlib) [ 2428.632612] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 2428.722355] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 2430.303180] Key type lgssc registered [ 2430.649330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2433.144451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2433.152352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2434.306198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2435.633425] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2438.354811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2438.469394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2439.734844] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2441.949789] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2443.678130] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2445.863810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2445.981036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2447.926981] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2450.247499] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2460.328967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2473.951918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2473.954374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2473.959317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2476.000512] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2476.002546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.003099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2476.007367] Lustre: Skipped 1 previous similar message [ 2478.344277] Lustre: server umount lustre-OST0000 complete [ 2479.583898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2479.586084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.590073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2485.721380] Lustre: server umount lustre-MDT0000 complete [ 2486.239957] LustreError: 184100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2486.244231] LustreError: 184100:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2486.871065] LustreError: 184080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663324 with bad export cookie 10001285411404340161 [ 2486.874898] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2486.876379] LustreError: 184080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2486.986831] Lustre: server umount lustre-MDT0001 complete [ 2490.313527] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 2491.259854] Key type lgssc unregistered [ 2491.372348] LNet: 187651:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2491.376598] LNetError: 187651:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2491.386259] LNet: Removed LNI 192.168.204.136@tcp [ 2491.674568] Key type .llcrypt unregistered [ 2491.675575] Key type ._llcrypt unregistered [ 2500.030321] Key type ._llcrypt registered [ 2500.032011] Key type .llcrypt registered [ 2500.067941] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2500.363258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2500.391030] alg: No test for adler32 (adler32-zlib) [ 2501.227594] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 2501.300667] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 2502.871143] Key type lgssc registered [ 2503.209820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2505.525350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2505.532877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2506.682406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2507.950257] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2510.660382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2510.781717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2512.011978] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2514.274814] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2515.998806] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2518.081591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2518.190310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2519.995179] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2522.293661] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2531.493531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2536.928887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2536.930398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2536.931450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2536.932466] LustreError: Skipped 1 previous similar message [ 2536.937947] Lustre: Skipped 1 previous similar message [ 2538.093244] Lustre: lustre-OST0000: Not available for connect from 192.168.204.36@tcp (stopping) [ 2538.095384] Lustre: Skipped 1 previous similar message [ 2541.197242] Lustre: server umount lustre-OST0000 complete [ 2543.450806] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8c03c3836000.ost_server_uuid 50 [ 2543.967064] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2556.896081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2556.898341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2556.902636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2560.098832] Lustre: server umount lustre-MDT0000 complete [ 2561.330380] LustreError: 189776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663398 with bad export cookie 4009970044730190566 [ 2561.333822] LustreError: 189776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2561.333936] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2561.462555] Lustre: server umount lustre-MDT0001 complete [ 2564.309370] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 22:30:00 (1781663400) [ 2566.375776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2566.516444] LustreError: 192955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2566.521515] LustreError: 192955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2566.539063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2567.795108] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2570.475646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2570.593663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2571.838452] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2574.057074] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2575.760876] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2577.900817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2579.866945] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2582.141551] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2584.278065] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2584.353780] Lustre: lustre-OST0001: new disk, initializing [ 2584.355276] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2584.357452] Lustre: 195793:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2584.387669] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2584.390769] Lustre: Skipped 1 previous similar message [ 2586.237990] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2587.584361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2587.588806] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2587.613066] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2589.755560] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2590.687780] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2590.690090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2590.693401] Lustre: Skipped 1 previous similar message [ 2590.695436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2590.697766] Lustre: Skipped 4 previous similar messages [ 2594.925675] Lustre: lustre-OST0000: Not available for connect from 192.168.204.36@tcp (stopping) [ 2594.927893] Lustre: Skipped 2 previous similar messages [ 2596.667906] Lustre: server umount lustre-OST0000 complete [ 2596.835780] LustreError: 194775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.842825] LustreError: 194775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2598.520377] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2598.585924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2599.875533] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2599.940565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2600.039701] LustreError: 195166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2605.160153] LustreError: 194775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2605.167613] LustreError: 194775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2612.194307] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2612.194700] LustreError: 195166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2612.195663] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2612.197901] Lustre: Skipped 1 previous similar message [ 2612.204915] LustreError: 195166:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2615.949934] Lustre: server umount lustre-OST0001 complete [ 2617.312546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2617.318825] Lustre: Skipped 1 previous similar message [ 2623.319838] Lustre: server umount lustre-MDT0000 complete [ 2624.507553] LustreError: 192936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663461 with bad export cookie 4009970044730191623 [ 2624.510882] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2624.512840] LustreError: 192936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2624.632290] Lustre: server umount lustre-MDT0001 complete [ 2632.130366] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 22:31:08 (1781663468) [ 2634.214480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2634.307785] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2634.318429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2634.389139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2635.695207] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2638.370824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2638.402242] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2638.418697] Lustre: 199212:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2639.842184] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2642.115543] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2643.822077] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2645.995925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2646.065602] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2648.019104] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2650.374814] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2652.649031] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2652.678721] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2652.842926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2652.856330] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 2654.718569] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2657.092538] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2658.274789] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2658.278100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2658.278381] LustreError: Skipped 1 previous similar message [ 2658.279262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2658.279265] Lustre: Skipped 5 previous similar messages [ 2658.282597] Lustre: Skipped 1 previous similar message [ 2664.059964] Lustre: server umount lustre-OST0000 complete [ 2667.624385] LustreError: 202111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.753328] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2678.755304] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2678.760503] Lustre: Skipped 1 previous similar message [ 2680.721927] Lustre: server umount lustre-OST0001 complete [ 2688.155269] Lustre: server umount lustre-MDT0000 complete [ 2688.992090] LustreError: 199222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2688.996755] LustreError: 199222:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2689.329327] LustreError: 201081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663526 with bad export cookie 4009970044730192785 [ 2689.332501] LustreError: 201081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2689.332528] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2689.456270] Lustre: server umount lustre-MDT0001 complete [ 2697.285626] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 22:32:13 (1781663533) [ 2702.333264] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 2705.254678] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2716.017731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2716.102730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2716.114469] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2716.150534] Lustre: lustre-MDT0000: new disk, initializing [ 2716.175039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2716.178052] Lustre: Skipped 3 previous similar messages [ 2716.183795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2717.520704] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2721.498267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2721.538037] Lustre: 207266:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2721.562711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2721.565629] Lustre: Skipped 1 previous similar message [ 2721.609794] Lustre: lustre-MDT0001: new disk, initializing [ 2721.653233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2721.658484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2722.927726] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2726.457090] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2728.133315] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2730.247569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2730.349597] Lustre: lustre-OST0000: new disk, initializing [ 2730.351773] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2730.354113] Lustre: 209235:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2732.302457] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2735.878607] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2736.122805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2736.126364] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2736.158721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2737.338398] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2737.416223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2738.841191] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2738.915288] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2741.218675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.219028] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2741.222442] Lustre: Skipped 2 previous similar messages [ 2741.225766] LustreError: Skipped 2 previous similar messages [ 2741.226413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2741.231990] Lustre: Skipped 9 previous similar messages [ 2745.680681] Lustre: server umount lustre-OST0000 complete [ 2761.183098] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2761.240167] Lustre: server umount lustre-MDT0000 complete [ 2761.696213] LustreError: 207279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2761.701894] LustreError: 207279:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2762.451360] LustreError: 207258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663599 with bad export cookie 4009970044730194451 [ 2762.454538] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2762.454799] LustreError: 207258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2762.581697] Lustre: server umount lustre-MDT0001 complete [ 2765.306984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2766.769961] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2769.479970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2770.909044] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2773.175328] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2774.849952] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2776.957702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2778.816740] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2781.087987] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2782.409798] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2790.619895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2791.953822] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2792.035022] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2794.463846] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2794.466098] LustreError: Skipped 1 previous similar message [ 2798.796594] Lustre: server umount lustre-OST0000 complete [ 2800.595899] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2801.795983] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2825.234022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2825.333520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2825.337282] Lustre: Skipped 5 previous similar messages [ 2827.225990] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2829.543397] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2830.101219] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2833.171583] Lustre: server umount lustre-OST0000 complete [ 2835.937172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2835.938436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2835.942608] Lustre: Skipped 5 previous similar messages [ 2835.948502] Lustre: Skipped 9 previous similar messages [ 2836.570562] Lustre: server umount lustre-MDT0000 complete [ 2837.801118] LustreError: 211427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663675 with bad export cookie 4009970044730195319 [ 2837.804333] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2837.806571] LustreError: 211427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2837.931906] Lustre: server umount lustre-MDT0001 complete [ 2840.578942] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 22:34:37 (1781663677) [ 2842.610491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2842.750442] LustreError: 217459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2842.756665] LustreError: 217459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2844.018323] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2846.742528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2848.160430] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2850.538303] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2852.422757] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2854.683689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2856.621911] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2859.016706] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2860.322755] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2868.563494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2870.021684] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2870.090536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2872.425679] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2872.500685] Lustre: lustre-OST0001: new disk, initializing [ 2872.502597] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2872.505298] Lustre: 220881:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2874.423592] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2878.108578] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2879.563737] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2882.047313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2882.050537] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2882.081818] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2882.698491] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2883.947172] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2884.009415] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2887.137311] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2887.141248] LustreError: Skipped 2 previous similar messages [ 2890.685872] Lustre: server umount lustre-OST0001 complete [ 2892.598470] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2892.675049] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2893.928557] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2893.994070] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2917.534187] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2919.521923] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2921.808448] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2922.328472] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2929.016170] Lustre: server umount lustre-OST0001 complete [ 2955.231092] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2955.282233] Lustre: server umount lustre-OST0000 complete [ 2962.779553] Lustre: server umount lustre-MDT0000 complete [ 2964.016144] LustreError: 217439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663801 with bad export cookie 4009970044730196250 [ 2964.020125] LustreError: 217439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2964.020287] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2970.223800] Lustre: server umount lustre-MDT0001 complete [ 2977.543076] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 22:36:53 (1781663813) [ 2979.659526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2979.752750] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2979.764558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2979.839292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2979.841302] Lustre: Skipped 5 previous similar messages [ 2981.136941] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2983.874104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.912437] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2983.923730] Lustre: 226139:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2985.321280] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2987.677939] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2989.400538] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2991.539089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2991.608496] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2993.504945] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2995.818330] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3004.935322] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3004.959163] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3006.933740] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3009.208492] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3009.725961] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3011.066424] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3013.182930] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3014.479445] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3014.546188] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3016.347954] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c03d0840800.ost_server_uuid 50 [ 3016.826533] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8c03d0840800.ost_server_uuid in FULL state after 0 sec [ 3017.163736] Lustre: Permanently deactivating lustre-OST0001 [ 3017.168733] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3017.947386] Lustre: Permanently reactivating lustre-OST0001 [ 3022.820513] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3022.820826] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3022.823779] LustreError: Skipped 1 previous similar message [ 3022.827558] Lustre: Skipped 6 previous similar messages [ 3022.828151] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3022.832127] Lustre: Skipped 18 previous similar messages [ 3025.023141] Lustre: server umount lustre-OST0001 complete [ 3027.939917] LustreError: 227984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3027.944481] LustreError: 227984:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 28 previous similar messages [ 3032.339424] Lustre: server umount lustre-OST0000 complete [ 3039.766625] Lustre: server umount lustre-MDT0000 complete [ 3040.957086] LustreError: 226174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663878 with bad export cookie 4009970044730197510 [ 3040.959980] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3040.961759] LustreError: 226174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3041.085106] Lustre: server umount lustre-MDT0001 complete [ 3048.655462] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 22:38:05 (1781663885) [ 3051.070960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3051.158393] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3051.167735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3051.170934] Lustre: Skipped 5 previous similar messages [ 3052.491755] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3055.185968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3055.217722] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3055.225978] Lustre: 232835:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3056.607503] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3058.865433] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3060.569444] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3062.673756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3062.749263] Lustre: Permanently deactivating lustre-OST0000 [ 3062.753944] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3064.628834] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3066.928979] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3067.381241] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3069.151649] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3071.191659] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3073.675671] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3074.072382] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3079.698289] Lustre: Permanently reactivating lustre-OST0000 [ 3079.704625] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 3079.708036] Lustre: Skipped 2 previous similar messages [ 3081.077712] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3081.080387] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3081.083842] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3081.100302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3081.108137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3090.748314] Lustre: server umount lustre-OST0001 complete [ 3106.271125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3106.323171] Lustre: server umount lustre-OST0000 complete [ 3113.762223] Lustre: server umount lustre-MDT0000 complete [ 3114.996156] LustreError: 232827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781663952 with bad export cookie 4009970044730199148 [ 3115.000054] LustreError: 232827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3115.000131] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3115.123251] Lustre: server umount lustre-MDT0001 complete [ 3118.655707] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 3119.648427] Key type lgssc unregistered [ 3119.770275] LNet: 237687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3119.773769] LNetError: 237687:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3119.782326] LNet: Removed LNI 192.168.204.136@tcp [ 3120.075758] Key type .llcrypt unregistered [ 3120.076729] Key type ._llcrypt unregistered [ 3128.738297] Key type ._llcrypt registered [ 3128.739745] Key type .llcrypt registered [ 3128.777443] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3129.159422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3129.187964] alg: No test for adler32 (adler32-zlib) [ 3130.041149] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 3130.132916] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 3131.719131] Key type lgssc registered [ 3132.074551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3135.275915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3135.283412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3136.440734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3137.720231] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3140.468217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3140.513974] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3140.524946] Lustre: 239119:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3140.528595] Lustre: Permanently deactivating lustre-MDT0001 [ 3140.534136] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3140.638098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3141.355063] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3141.357145] LustreError: 239159:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3141.360495] LustreError: 239159:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3141.363625] LustreError: 239159:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 1, failed: rc = -108 [ 3141.964639] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3144.227172] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3145.956184] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3148.189592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3148.304733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3150.197874] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3150.371360] LustreError: 241351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3150.375097] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3150.376889] LustreError: 241351:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3152.522425] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3154.785926] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3154.861093] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3156.698793] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3158.991332] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3160.038657] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3161.065081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3161.065427] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3164.692305] Lustre: Permanently reactivating lustre-MDT0001 [ 3164.698066] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3164.700247] Lustre: Skipped 2 previous similar messages [ 3167.725341] Lustre: lustre-MDT0001: Client 8ea8189a-f398-4c3d-8cb9-e39ffc84ddb0 (at 192.168.204.36@tcp) reconnecting [ 3169.528986] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3170.033182] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3171.801901] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3171.877161] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3173.436042] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3173.504444] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3174.285226] Lustre: Permanently deactivating lustre-MDT0001 [ 3174.290658] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3174.292599] Lustre: Skipped 2 previous similar messages [ 3179.517801] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3183.586374] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3183.587837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3183.590843] Lustre: Skipped 3 previous similar messages [ 3183.593699] Lustre: Skipped 3 previous similar messages [ 3188.257358] Lustre: server umount lustre-MDT0000 complete [ 3188.704797] LustreError: 239128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3188.709544] LustreError: 239128:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3189.525602] LustreError: 239111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781664026 with bad export cookie 9051013366460843030 [ 3189.530063] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3189.530467] LustreError: 239111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3189.654720] Lustre: server umount lustre-MDT0001 complete [ 3196.984888] Lustre: server umount lustre-OST0000 complete [ 3200.504862] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 22:40:36 (1781664036) [ 3204.058322] Lustre: server umount lustre-OST0001 complete [ 3206.378506] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 3209.286198] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3220.556741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3220.639721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3220.642648] Lustre: Skipped 2 previous similar messages [ 3220.651473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3220.687434] Lustre: lustre-MDT0000: new disk, initializing [ 3220.709451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3220.715632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3222.055228] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3226.111434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3226.152461] Lustre: 247421:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3226.177189] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3226.179192] Lustre: Skipped 1 previous similar message [ 3226.214312] Lustre: lustre-MDT0001: new disk, initializing [ 3226.253616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3226.256704] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3227.580552] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3231.197966] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3232.917148] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3235.088253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3235.191625] Lustre: lustre-OST0000: new disk, initializing [ 3235.193142] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3235.196486] Lustre: 249389:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3235.221828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3235.223857] Lustre: Skipped 1 previous similar message [ 3237.071533] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3240.650862] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3242.486982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3242.490199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3242.519936] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3246.688910] LustreError: 247430:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3250.940594] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3251.021470] Lustre: lustre-OST0001: new disk, initializing [ 3251.023817] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3251.027992] Lustre: 250503:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3252.964673] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3256.583601] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3256.783139] LustreError: 247430:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3256.793569] LustreError: 247428:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3258.878111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3258.881952] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3258.910315] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3260.177605] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 3266.887112] LustreError: 247428:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3269.091724] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3269.092046] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3269.093104] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3269.097269] Lustre: Skipped 1 previous similar message [ 3269.099318] LustreError: Skipped 1 previous similar message [ 3273.597626] Lustre: server umount lustre-OST0001 complete [ 3274.212651] LustreError: 249374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3274.220637] LustreError: 249374:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3279.330774] LustreError: 251224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3279.335473] LustreError: 251224:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3284.453484] LustreError: 249373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3284.460874] LustreError: 249373:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3288.544227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3288.546799] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3288.551823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3288.553701] Lustre: Skipped 3 previous similar messages [ 3291.733756] Lustre: server umount lustre-OST0000 complete [ 3293.152072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3293.155799] LustreError: Skipped 1 previous similar message [ 3293.158034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3293.163990] Lustre: Skipped 1 previous similar message [ 3293.166634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3293.169498] Lustre: Skipped 2 previous similar messages [ 3299.232502] Lustre: server umount lustre-MDT0000 complete [ 3299.808058] LustreError: 247434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3299.813026] LustreError: 247434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3300.488171] LustreError: 248216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781664137 with bad export cookie 9051013366460845844 [ 3300.491017] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3300.492990] LustreError: 248216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3306.683315] Lustre: server umount lustre-MDT0001 complete [ 3310.225753] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 3311.213467] Key type lgssc unregistered [ 3311.333342] LNet: 252534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3311.335788] LNetError: 252534:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3311.344298] LNet: Removed LNI 192.168.204.136@tcp [ 3311.629149] Key type .llcrypt unregistered [ 3311.631079] Key type ._llcrypt unregistered [ 3326.314469] Key type ._llcrypt registered [ 3326.315553] Key type .llcrypt registered [ 3326.349311] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3326.828676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3326.833285] alg: No test for adler32 (adler32-zlib) [ 3327.682480] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 3327.764615] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 3329.343226] Key type lgssc registered [ 3329.833992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3332.442905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3332.449636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3333.543165] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3333.550201] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3333.608768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3334.937359] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3337.655715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3337.688624] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3337.698333] Lustre: 254719:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3337.785156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3339.084641] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3341.518989] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3343.328211] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3345.567813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3345.637199] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3345.700512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3347.698922] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3347.924593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3350.032407] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3388.897345] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3388.898715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3388.899560] LustreError: Skipped 1 previous similar message [ 3388.907069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3388.909449] Lustre: Skipped 1 previous similar message [ 3394.016959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3394.020286] Lustre: Skipped 1 previous similar message [ 3394.777443] Lustre: server umount lustre-OST0000 complete [ 3396.210471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3400.257034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3400.377990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3402.237832] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3404.494417] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3418.723708] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3418.727149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3418.727155] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3418.731793] Lustre: Skipped 1 previous similar message [ 3439.586140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3439.586253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3439.589765] Lustre: Skipped 1 previous similar message [ 3439.591956] LustreError: Skipped 1 previous similar message [ 3439.592476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3439.596638] Lustre: Skipped 1 previous similar message [ 3443.352270] Lustre: server umount lustre-OST0000 complete [ 3444.704392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3444.705028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3444.708819] Lustre: Skipped 1 previous similar message [ 3449.824671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3449.828819] Lustre: Skipped 2 previous similar messages [ 3450.776385] Lustre: server umount lustre-MDT0000 complete [ 3452.161719] LustreError: 255460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781664289 with bad export cookie 11895221540618381052 [ 3452.164842] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3452.165116] LustreError: 255460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3452.318940] Lustre: server umount lustre-MDT0001 complete [ 3455.857239] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 3456.879749] Key type lgssc unregistered [ 3457.010508] LNet: 260128:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3457.014311] LNetError: 260128:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3457.021507] LNet: Removed LNI 192.168.204.136@tcp [ 3457.379591] Key type .llcrypt unregistered [ 3457.381323] Key type ._llcrypt unregistered [ 3466.111161] Key type ._llcrypt registered [ 3466.112214] Key type .llcrypt registered [ 3466.158424] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3466.545984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3466.573517] alg: No test for adler32 (adler32-zlib) [ 3467.436787] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 3467.516821] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 3469.103129] Key type lgssc registered [ 3469.459126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3472.115411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3472.122524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3473.311868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3474.649566] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3477.508069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3477.631486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3478.993071] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3481.333450] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3483.081018] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3485.236085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3485.353377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3487.246482] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3489.812598] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3498.665460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3508.191912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3508.195338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3508.200313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3509.217117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3509.217175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3509.223094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3513.490050] Lustre: server umount lustre-OST0000 complete [ 3518.943896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3518.946203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3518.949895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3518.951519] Lustre: Skipped 1 previous similar message [ 3524.576102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3524.577847] Lustre: Skipped 3 previous similar messages [ 3529.183137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3529.248770] Lustre: server umount lustre-MDT0000 complete [ 3529.695930] LustreError: 262954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3529.700861] LustreError: 262954:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3530.451164] LustreError: 261501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781664367 with bad export cookie 13161291911956526083 [ 3530.451823] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3530.454489] LustreError: 261501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3530.563642] Lustre: server umount lustre-MDT0001 complete [ 3533.985627] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 3534.963570] Key type lgssc unregistered [ 3535.097613] LNet: 265850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3535.101242] LNetError: 265850:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3535.109248] LNet: Removed LNI 192.168.204.136@tcp [ 3535.397620] Key type .llcrypt unregistered [ 3535.398648] Key type ._llcrypt unregistered [ 3542.223447] Key type ._llcrypt registered [ 3542.225183] Key type .llcrypt registered [ 3542.260419] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3542.635200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3542.653572] alg: No test for adler32 (adler32-zlib) [ 3543.502420] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 3543.581743] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 3545.159195] Key type lgssc registered [ 3545.531956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3548.186455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3548.192086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3549.345431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3550.634984] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3553.354549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3553.468947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3554.772457] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3557.072675] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3558.842459] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3561.025424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3561.148553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3563.066894] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3565.484666] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3575.526373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3580.898351] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3580.898879] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3580.899885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3580.904294] Lustre: Skipped 1 previous similar message [ 3581.388293] Lustre: server umount lustre-OST0000 complete [ 3584.479798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3584.482178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3584.487241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3584.490203] Lustre: Skipped 1 previous similar message [ 3584.663408] Lustre: server umount lustre-MDT0000 complete [ 3585.866578] LustreError: 267029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781664423 with bad export cookie 12697627983155537510 [ 3585.869875] LustreError: 267029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3585.870347] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3585.994687] Lustre: server umount lustre-MDT0001 complete [ 3589.483372] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 3590.531332] Key type lgssc unregistered [ 3590.661502] LNet: 270741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3590.664138] LNetError: 270741:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3590.672426] LNet: Removed LNI 192.168.204.136@tcp [ 3590.984047] Key type .llcrypt unregistered [ 3590.985289] Key type ._llcrypt unregistered [ 3600.021096] Key type ._llcrypt registered [ 3600.022220] Key type .llcrypt registered [ 3600.057365] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3600.353917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3600.393064] alg: No test for adler32 (adler32-zlib) [ 3601.242814] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 3601.320157] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 3602.895110] Key type lgssc registered [ 3603.261770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3605.729076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3605.736156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3606.897794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3608.192639] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3610.925824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3611.053520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3612.356486] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3614.661383] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3616.449351] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3618.596338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3618.708443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3620.535181] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3622.753700] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3632.039965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3641.312131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3641.316587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3641.326854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3642.337325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3642.338631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3642.339384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3642.347189] Lustre: Skipped 1 previous similar message [ 3646.606411] Lustre: server umount lustre-OST0000 complete [ 3652.063812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3652.066039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3652.069661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3657.695987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3657.698806] Lustre: Skipped 2 previous similar messages [ 3662.303089] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3662.358896] Lustre: server umount lustre-MDT0000 complete [ 3662.817219] LustreError: 274432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3662.821276] LustreError: 274432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3663.581254] LustreError: 273976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781664500 with bad export cookie 10720685960143019929 [ 3663.584672] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3663.587761] LustreError: 273976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3663.706451] Lustre: server umount lustre-MDT0001 complete [ 3667.224506] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 3668.212846] Key type lgssc unregistered [ 3668.344337] LNet: 276460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3668.346870] LNetError: 276460:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3668.356296] LNet: Removed LNI 192.168.204.136@tcp [ 3668.660556] Key type .llcrypt unregistered [ 3668.661616] Key type ._llcrypt unregistered [ 3675.452263] Key type ._llcrypt registered [ 3675.453436] Key type .llcrypt registered [ 3675.494850] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3676.000911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3676.017240] alg: No test for adler32 (adler32-zlib) [ 3676.868905] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 3676.953645] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 3678.527128] Key type lgssc registered [ 3678.873729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3681.247779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3681.253443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3682.416930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3683.648653] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3686.191169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3686.302385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3687.557933] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3689.765185] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3691.654946] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3693.964488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3694.074668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3696.034902] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3698.302827] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3707.366705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3712.481572] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3712.483965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3712.488816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3712.490516] Lustre: Skipped 1 previous similar message [ 3717.600374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3717.602381] Lustre: Skipped 1 previous similar message [ 3718.603419] Lustre: server umount lustre-OST0000 complete [ 3722.208395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3722.211954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3722.216023] Lustre: Skipped 1 previous similar message [ 3722.217947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3725.971780] Lustre: server umount lustre-MDT0000 complete [ 3727.180230] LustreError: 280351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781664564 with bad export cookie 4228550271977342169 [ 3727.183459] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3727.184082] LustreError: 280351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3727.306763] Lustre: server umount lustre-MDT0001 complete [ 3730.804356] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 3731.739703] Key type lgssc unregistered [ 3731.857311] LNet: 281388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3731.860513] LNetError: 281388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3731.869295] LNet: Removed LNI 192.168.204.136@tcp [ 3732.172663] Key type .llcrypt unregistered [ 3732.173979] Key type ._llcrypt unregistered [ 3735.680026] Key type ._llcrypt registered [ 3735.681559] Key type .llcrypt registered [ 3735.716930] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 3741.016664] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 3746.827814] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3747.242920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3747.249400] alg: No test for adler32 (adler32-zlib) [ 3748.096550] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 3748.184508] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 3749.759161] Key type lgssc registered [ 3750.154385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3761.461448] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3761.466275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3762.556506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3762.569374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3762.604973] Lustre: lustre-MDT0000: new disk, initializing [ 3762.630543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3762.638038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3764.295188] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3769.122344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3769.174567] Lustre: 284911:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3769.202242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3769.205056] Lustre: Skipped 1 previous similar message [ 3769.246636] Lustre: lustre-MDT0001: new disk, initializing [ 3769.281492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3769.292408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3769.295422] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3770.712014] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3774.226569] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3775.882613] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3777.980895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3778.093987] Lustre: lustre-OST0000: new disk, initializing [ 3778.095725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3778.098537] Lustre: 286879:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3778.127787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3779.973183] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3783.683617] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3784.212829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3784.218554] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3784.261386] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3785.137628] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3785.202963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3786.519515] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3786.592522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3789.282936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3789.283625] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3789.284212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3789.286328] Lustre: Skipped 1 previous similar message [ 3789.297390] LustreError: Skipped 1 previous similar message [ 3793.360117] Lustre: server umount lustre-OST0000 complete [ 3794.911809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3794.915051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3794.921418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3794.923736] Lustre: Skipped 1 previous similar message [ 3799.520067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3799.520645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3804.640248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3804.641950] Lustre: Skipped 2 previous similar messages [ 3808.736319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3808.738023] Lustre: Skipped 1 previous similar message [ 3809.247071] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3809.309100] Lustre: server umount lustre-MDT0000 complete [ 3810.491680] LustreError: 284905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781664647 with bad export cookie 12329356121548481840 [ 3810.495377] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3810.497444] LustreError: 284905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3810.608142] Lustre: server umount lustre-MDT0001 complete [ 3813.184769] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 22:50:49 (1781664649) [ 3815.332695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3815.516780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3816.810210] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3819.888295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3820.010212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3821.314482] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3823.687629] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3825.450151] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3827.620577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3829.561118] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3831.947346] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3850.209067] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3850.209662] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.211313] LustreError: Skipped 1 previous similar message [ 3850.215409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3850.217670] Lustre: Skipped 1 previous similar message [ 3851.761099] Lustre: server umount lustre-OST0000 complete [ 3854.817471] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3854.819966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3854.823401] Lustre: Skipped 1 previous similar message [ 3859.284414] Lustre: server umount lustre-MDT0000 complete [ 3860.449133] LustreError: 289291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3860.455336] LustreError: 289291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3860.479582] LustreError: 289270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781664697 with bad export cookie 12329356121548482694 [ 3860.482266] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3860.483082] LustreError: 289270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3860.590553] Lustre: server umount lustre-MDT0001 complete [ 3864.552193] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 3865.893853] Key type lgssc unregistered [ 3866.049567] LNet: 292845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3866.052429] LNetError: 292845:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3866.063625] LNet: Removed LNI 192.168.204.136@tcp [ 3866.410144] Key type .llcrypt unregistered [ 3866.412065] Key type ._llcrypt unregistered [ 3876.211129] Key type ._llcrypt registered [ 3876.212622] Key type .llcrypt registered [ 3876.252897] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 3882.220480] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3882.574536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3882.637412] alg: No test for adler32 (adler32-zlib) [ 3883.508742] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 3883.603806] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 3885.199112] Key type lgssc registered [ 3885.549932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3900.010323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3900.016989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3901.105035] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3901.116915] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3901.152147] Lustre: lustre-MDT0000: new disk, initializing [ 3901.173853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3901.179492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3902.479854] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3906.634421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3906.692458] Lustre: 296641:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3906.715365] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3906.717089] Lustre: Skipped 1 previous similar message [ 3906.750907] Lustre: lustre-MDT0001: new disk, initializing [ 3906.780929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3906.792562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3906.797827] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3908.126213] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3911.725392] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3913.585800] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3915.964454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3916.077871] Lustre: lustre-OST2710: new disk, initializing [ 3916.079542] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 3916.081940] Lustre: 298610:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3916.116701] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 3917.303585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 3917.306930] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 3917.339676] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 3918.206642] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3921.878816] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 3924.052662] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3924.122902] Lustre: lustre-OST03e8: new disk, initializing [ 3924.124701] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 3924.127210] Lustre: 299673:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3924.160431] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 3924.858457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 3924.861552] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 3924.891681] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 3926.027477] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3928.365823] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 3930.953604] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 3931.017202] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3932.367485] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 3932.432061] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3933.777075] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 3933.853350] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3935.192867] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 3935.267655] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3937.760377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3937.762578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3937.766404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3940.321082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3940.323177] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3940.330235] Lustre: Skipped 2 previous similar messages [ 3942.429850] Lustre: server umount lustre-MDT0000 complete [ 3943.651155] LustreError: 296632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781664780 with bad export cookie 10836227355535154016 [ 3943.654250] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3943.655910] LustreError: 296632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3943.770854] Lustre: server umount lustre-MDT0001 complete [ 3955.656390] Lustre: server umount lustre-OST2710 complete [ 3967.204228] Lustre: server umount lustre-OST03e8 complete [ 3973.862763] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 3977.191141] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3990.844479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3990.958876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3990.971999] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3991.016593] Lustre: lustre-MDT0000: new disk, initializing [ 3991.039752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3991.046170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3992.406802] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3996.354507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3996.393161] Lustre: 304913:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3996.491469] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3997.798615] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4001.409725] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4003.145320] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4005.288190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4005.390827] Lustre: lustre-OST0000: new disk, initializing [ 4005.392170] Lustre: Skipped 1 previous similar message [ 4005.393779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4005.395685] Lustre: Skipped 2 previous similar messages [ 4005.398106] Lustre: 306879:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4005.437318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4005.439206] Lustre: Skipped 1 previous similar message [ 4007.187656] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4010.474737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4010.477751] Lustre: Skipped 1 previous similar message [ 4010.479798] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4010.495236] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4010.696686] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4012.002393] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4012.071511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4013.467473] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4013.551400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4015.585840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4015.586879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4015.589757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4015.594568] Lustre: Skipped 1 previous similar message [ 4015.597509] Lustre: Skipped 3 previous similar messages [ 4016.269143] Lustre: server umount lustre-OST0000 complete [ 4020.704673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4020.705446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4020.709054] Lustre: Skipped 1 previous similar message [ 4020.711777] Lustre: Skipped 2 previous similar messages [ 4023.577971] Lustre: server umount lustre-MDT0000 complete [ 4024.757995] LustreError: 304905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781664862 with bad export cookie 10836227355535155682 [ 4024.761820] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4024.763331] LustreError: 304905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4024.872733] Lustre: server umount lustre-MDT0001 complete [ 4027.467811] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 22:54:23 (1781664863) [ 4027.930362] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4028.442785] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 22:54:24 (1781664864) [ 4029.421817] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4037.871385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4037.964498] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4037.972226] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4038.029712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4039.340457] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4042.092202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4042.131032] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4042.141504] Lustre: 310731:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4043.503151] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4059.234207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4059.301627] Lustre: Denying initial registration attempt from nid 192.168.204.136@tcp, specified as failover [ 4059.304279] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4059.308282] LustreError: 312179:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 4059.311562] LustreError: 312179:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 4059.314463] LustreError: 312179:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4059.335659] Lustre: server umount lustre-OST0000 complete [ 4059.336969] LustreError: 312179:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4062.687914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4062.691129] LustreError: Skipped 1 previous similar message [ 4062.693031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4062.698985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4067.618136] Lustre: server umount lustre-MDT0000 complete [ 4068.849180] LustreError: 311471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781664906 with bad export cookie 10836227355535156543 [ 4068.851497] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4068.854873] LustreError: 311471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4068.972514] Lustre: server umount lustre-MDT0001 complete [ 4072.357171] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 4073.332308] Key type lgssc unregistered [ 4073.449341] LNet: 313303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4073.451541] LNetError: 313303:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4073.459268] LNet: Removed LNI 192.168.204.136@tcp [ 4073.750678] Key type .llcrypt unregistered [ 4073.752346] Key type ._llcrypt unregistered [ 4077.151631] Key type ._llcrypt registered [ 4077.152768] Key type .llcrypt registered [ 4077.189185] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4077.505383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4077.544805] alg: No test for adler32 (adler32-zlib) [ 4078.395327] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 4078.476152] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 4080.055126] Key type lgssc registered [ 4080.408042] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4103.744772] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4106.448475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4106.456700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4107.547895] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4107.554549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4107.610711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4108.891942] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4112.279155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4112.309071] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4112.319170] Lustre: 316041:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4112.417227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4113.827371] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4116.285410] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4118.578603] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4134.178179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4134.252581] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4134.327467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4136.283376] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4138.850035] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4139.765486] Lustre: server umount lustre-OST0000 complete [ 4143.072420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4143.076121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4143.082844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4148.192615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4148.194950] Lustre: Skipped 2 previous similar messages [ 4153.313097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4153.317098] Lustre: Skipped 1 previous similar message [ 4155.359089] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4155.426902] Lustre: server umount lustre-MDT0000 complete [ 4156.755597] LustreError: 316031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781664994 with bad export cookie 7849341838058294878 [ 4156.760371] LustreError: 316031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4156.760649] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4156.880486] Lustre: server umount lustre-MDT0001 complete [ 4161.081931] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 4162.354868] Key type lgssc unregistered [ 4162.494551] LNet: 319606:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4162.499538] LNetError: 319606:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4162.512573] LNet: Removed LNI 192.168.204.136@tcp [ 4162.940163] Key type .llcrypt unregistered [ 4162.942143] Key type ._llcrypt unregistered [ 4173.063104] Key type ._llcrypt registered [ 4173.064178] Key type .llcrypt registered [ 4173.103848] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4173.430067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4173.436340] alg: No test for adler32 (adler32-zlib) [ 4174.300562] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 4174.403168] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 4176.008138] Key type lgssc registered [ 4176.383396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4179.303172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4179.313153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4180.504290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4182.038031] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4185.430432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4185.557120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4187.046112] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4190.344384] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4192.817506] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4198.115241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4198.286012] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4200.454151] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4203.642090] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4220.532355] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.36@tcp (stopping) [ 4220.901115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4220.909214] Lustre: Skipped 2 previous similar messages [ 4224.674838] Lustre: server umount lustre-MDT0000 complete [ 4225.640522] LustreError: 320994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4225.651602] LustreError: 320994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4226.182201] LustreError: 322842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781665063 with bad export cookie 3561442664299922528 [ 4226.188093] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4226.189713] LustreError: 322842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4230.758619] LustreError: 322294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4230.766352] LustreError: 322294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4230.766522] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.36@tcp (stopping) [ 4230.770503] Lustre: Skipped 3 previous similar messages [ 4231.136360] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4232.359271] Lustre: server umount lustre-MDT0001 complete [ 4234.751610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4238.056806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4251.616444] LustreError: 320310:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000044b4b173 x1868211170600448/t0(0) o250->MGC192.168.204.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4251.785925] LustreError: 322829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4251.792421] LustreError: 322829:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4251.812696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4251.835865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4253.244674] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4256.447076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4256.584676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4258.000504] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4260.686733] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4261.736588] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4266.542907] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4268.332385] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4270.949157] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4271.930399] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4276.195789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4276.197537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4276.201719] Lustre: Skipped 1 previous similar message [ 4276.203488] Lustre: Skipped 1 previous similar message [ 4279.265895] Lustre: server umount lustre-OST0000 complete [ 4281.312616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4281.313279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4281.317589] Lustre: Skipped 1 previous similar message [ 4281.319769] Lustre: Skipped 1 previous similar message [ 4286.432747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4286.436863] Lustre: Skipped 2 previous similar messages [ 4287.281234] Lustre: server umount lustre-MDT0000 complete [ 4289.091096] LustreError: 320980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781665126 with bad export cookie 3561442664299929689 [ 4289.096657] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4289.099229] LustreError: 320980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4289.252889] Lustre: server umount lustre-MDT0001 complete [ 4292.915182] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 4293.904365] Key type lgssc unregistered [ 4294.025308] LNet: 327283:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4294.027414] LNetError: 327283:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4294.035313] LNet: Removed LNI 192.168.204.136@tcp [ 4294.346977] Key type .llcrypt unregistered [ 4294.348151] Key type ._llcrypt unregistered [ 4303.292403] Key type ._llcrypt registered [ 4303.293570] Key type .llcrypt registered [ 4303.338911] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4303.661467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4303.727294] alg: No test for adler32 (adler32-zlib) [ 4304.576841] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 4304.654984] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 4306.231216] Key type lgssc registered [ 4306.697359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4309.907948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4309.918264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4311.082267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4312.369193] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4315.003710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4315.115325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4316.374155] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4318.571717] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4320.244462] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4321.210995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4321.212419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4321.214484] Lustre: Skipped 1 previous similar message [ 4325.344446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4325.346657] Lustre: Skipped 2 previous similar messages [ 4327.378901] Lustre: server umount lustre-MDT0000 complete [ 4328.551386] LustreError: 330292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781665165 with bad export cookie 6932096353042738251 [ 4328.554254] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4328.555476] LustreError: 330292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4328.684099] Lustre: server umount lustre-MDT0001 complete [ 4333.237043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4333.326740] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4333.337016] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4333.405609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4334.684324] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4337.379625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4337.415976] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4337.426955] Lustre: 331262:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4337.525162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4338.805320] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4340.991615] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4342.641744] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4346.978675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4347.045782] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4347.050435] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4347.053150] LustreError: 333143:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 4347.055329] LustreError: 333143:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 4347.057680] LustreError: 333143:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4347.078595] Lustre: server umount lustre-OST0000 complete [ 4347.079932] LustreError: 333143:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4351.778453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4351.843096] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4351.904950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4353.689893] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4355.880183] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4357.640925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4358.227787] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4358.257801] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4358.286831] Lustre: lustre-OST0001: new disk, initializing [ 4358.288447] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4358.291460] Lustre: 334682:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4360.213288] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4362.980627] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4364.510699] Lustre: server umount lustre-OST0001 complete [ 4365.832977] LustreError: 333610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4365.838795] LustreError: 333610:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4367.841513] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4367.841599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4367.846714] LustreError: Skipped 1 previous similar message [ 4367.847559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4367.854769] Lustre: Skipped 1 previous similar message [ 4370.916742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4370.918059] LustreError: 333607:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4370.918605] Lustre: Skipped 2 previous similar messages [ 4370.923107] LustreError: 333607:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4372.366057] Lustre: server umount lustre-OST0000 complete [ 4376.033933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4376.035662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4376.037529] Lustre: Skipped 1 previous similar message [ 4376.041081] Lustre: Skipped 1 previous similar message [ 4379.695578] Lustre: server umount lustre-MDT0000 complete [ 4381.099833] LustreError: 331253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781665218 with bad export cookie 6932096353042738790 [ 4381.106438] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4381.112669] LustreError: 331253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4381.248888] Lustre: server umount lustre-MDT0001 complete [ 4385.704460] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 4387.138710] Key type lgssc unregistered [ 4387.256338] LNet: 336636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4387.258502] LNetError: 336636:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4387.267329] LNet: Removed LNI 192.168.204.136@tcp [ 4387.545107] Key type .llcrypt unregistered [ 4387.546458] Key type ._llcrypt unregistered [ 4423.603387] Key type ._llcrypt registered [ 4423.604477] Key type .llcrypt registered [ 4423.650711] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 4430.035310] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4430.510827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4430.546978] alg: No test for adler32 (adler32-zlib) [ 4431.411441] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 4431.499305] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 4433.079194] Key type lgssc registered [ 4433.603767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4445.508891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4445.513602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4446.602248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4446.613061] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4446.653196] Lustre: lustre-MDT0000: new disk, initializing [ 4446.676171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4446.682735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4447.968089] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4452.473136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4452.522309] Lustre: 341550:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4452.555960] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4452.558655] Lustre: Skipped 1 previous similar message [ 4452.594933] Lustre: lustre-MDT0001: new disk, initializing [ 4452.632234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4452.646474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4452.652121] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4454.071873] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4457.842146] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4459.648720] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4462.214982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4462.346093] Lustre: lustre-OST0000: new disk, initializing [ 4462.347671] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4462.350043] Lustre: 343517:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4462.378909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4462.888825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4462.892866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4462.936184] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4464.658225] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4468.630263] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4469.899995] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4469.962313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4471.204659] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4471.265567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4473.314418] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4473.314642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4473.316483] LustreError: Skipped 1 previous similar message [ 4473.317149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4473.319556] Lustre: Skipped 1 previous similar message [ 4478.434971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4478.436778] Lustre: Skipped 2 previous similar messages [ 4483.561494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4483.567977] Lustre: Skipped 1 previous similar message [ 4486.111124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4486.169830] Lustre: server umount lustre-OST0000 complete [ 4488.674375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4488.674427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4488.675270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4488.678320] Lustre: Skipped 1 previous similar message [ 4493.541524] Lustre: server umount lustre-MDT0000 complete [ 4494.773218] LustreError: 341543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781665332 with bad export cookie 2187230101312870694 [ 4494.776841] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4494.779341] LustreError: 341543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4494.897320] Lustre: server umount lustre-MDT0001 complete [ 4497.386098] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 23:02:13 (1781665333) [ 4499.483503] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 23:02:15 (1781665335) [ 4501.952970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4502.095383] LustreError: 346271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4502.100509] LustreError: 346271:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4502.115161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4503.291773] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4506.822569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4506.980830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4508.373531] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4511.793627] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4514.030096] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4517.273441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4517.435761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4519.675595] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4522.591496] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4536.443421] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4537.059802] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4537.590408] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4538.105903] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4538.748537] Lustre: Failing over lustre-MDT0000 [ 4538.864923] Lustre: server umount lustre-MDT0000 complete [ 4539.501921] LustreError: 346266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4541.921107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4541.923273] LustreError: 348727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4541.927979] Lustre: Skipped 2 previous similar messages [ 4541.932242] LustreError: 348727:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4544.615353] LustreError: 348839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4544.624074] LustreError: 348839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4549.736428] LustreError: 348727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4549.741576] LustreError: 348727:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4551.708646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4551.748905] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4551.844564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4551.862193] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4553.149560] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4554.856380] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4557.285171] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4557.298238] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4557.316904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4559.001691] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4559.543850] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4561.385546] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4562.009720] Lustre: Failing over lustre-MDT0000 [ 4562.217036] Lustre: server umount lustre-MDT0000 complete [ 4562.399862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4562.402953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4562.409977] LustreError: 346998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4562.415630] LustreError: 346998:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4575.879945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4575.945640] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4576.098141] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4577.429145] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4580.457227] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4581.354391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4581.358970] Lustre: Skipped 2 previous similar messages [ 4581.373963] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4581.407610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4583.463429] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4584.259795] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4591.587536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4591.588845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4591.590709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4591.590720] Lustre: Skipped 1 previous similar message [ 4591.610374] Lustre: Skipped 4 previous similar messages [ 4593.072809] Lustre: server umount lustre-MDT0000 complete [ 4594.734108] LustreError: 346251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781665432 with bad export cookie 2187230101312873375 [ 4594.740096] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4594.743211] LustreError: 346251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4594.882505] Lustre: server umount lustre-MDT0001 complete [ 4607.036188] Lustre: server umount lustre-OST0000 complete [ 4613.507876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4613.703207] LustreError: 352618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4613.711785] LustreError: 352618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 4613.728862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4613.730799] Lustre: Skipped 1 previous similar message [ 4615.545473] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4618.932700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4620.808751] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4623.813756] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4626.477616] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4629.460138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4632.051534] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4634.658047] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4638.889418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4647.012531] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4647.014799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4647.024458] Lustre: Skipped 1 previous similar message [ 4647.028289] Lustre: Skipped 2 previous similar messages [ 4652.265208] Lustre: server umount lustre-OST0000 complete [ 4655.072083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4666.848332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4666.850359] Lustre: Skipped 7 previous similar messages [ 4668.383107] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4668.436745] Lustre: server umount lustre-MDT0000 complete [ 4669.577248] LustreError: 355417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781665506 with bad export cookie 2187230101312874096 [ 4669.580917] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4669.583367] LustreError: 355417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4669.698310] Lustre: server umount lustre-MDT0001 complete [ 4672.803188] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 4673.701216] Key type lgssc unregistered [ 4673.818283] LNet: 356168:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4673.820653] LNetError: 356168:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4673.830247] LNet: Removed LNI 192.168.204.136@tcp [ 4674.108892] Key type .llcrypt unregistered [ 4674.110957] Key type ._llcrypt unregistered [ 4683.512463] Key type ._llcrypt registered [ 4683.513507] Key type .llcrypt registered [ 4683.548020] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4683.988667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4683.998825] alg: No test for adler32 (adler32-zlib) [ 4684.851113] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 4684.941587] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 4686.527104] Key type lgssc registered [ 4686.831177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4689.162513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4689.167741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4690.313536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4691.556624] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4694.186511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4694.304683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4695.631148] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4698.041276] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4699.706900] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4701.853779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4701.969392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4703.758840] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4705.983977] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4715.302670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 4720.609439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4720.609968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4720.613272] Lustre: Skipped 1 previous similar message [ 4725.215925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4725.218026] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4725.221473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4725.223046] Lustre: Skipped 1 previous similar message [ 4726.421407] Lustre: server umount lustre-MDT0000 complete [ 4727.564042] LustreError: 357641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781665564 with bad export cookie 15479422554009589670 [ 4727.567434] LustreError: 357641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4727.567638] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4727.676708] Lustre: server umount lustre-MDT0001 complete [ 4739.100205] Lustre: server umount lustre-OST0000 complete [ 4744.114286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4744.260416] LustreError: 361390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4744.266213] LustreError: 361390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4744.281118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4745.802251] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4748.630924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4748.748369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4750.083676] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4752.375117] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4754.214964] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4756.402762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4758.388103] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4760.631137] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4768.806187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4778.977117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4778.978356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4778.982851] Lustre: Skipped 2 previous similar messages [ 4778.984603] Lustre: Skipped 3 previous similar messages [ 4783.768339] Lustre: server umount lustre-MDT0000 complete [ 4784.096233] LustreError: 362117:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4784.101723] LustreError: 362117:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4784.922259] LustreError: 362125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781665622 with bad export cookie 15479422554009590769 [ 4784.926521] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4784.927746] LustreError: 362125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4785.040199] Lustre: server umount lustre-MDT0001 complete [ 4796.505058] Lustre: server umount lustre-OST0000 complete [ 4801.366449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4801.572484] LustreError: 365319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4801.617375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4801.622234] Lustre: Skipped 1 previous similar message [ 4803.740208] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4806.627731] LustreError: 365320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4807.793388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4809.542040] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4812.654125] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4814.655577] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4817.584340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4817.740813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4817.742855] Lustre: Skipped 1 previous similar message [ 4820.439577] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4823.407708] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4825.965251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4831.206119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4831.207476] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4831.208070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4831.208080] Lustre: Skipped 1 previous similar message [ 4831.215128] Lustre: Skipped 1 previous similar message [ 4832.298363] Lustre: server umount lustre-OST0000 complete [ 4833.759824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4835.829183] Lustre: server umount lustre-MDT0000 complete [ 4836.320730] LustreError: 365320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4836.321533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4836.337172] Lustre: Skipped 1 previous similar message [ 4837.585299] LustreError: 366053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781665674 with bad export cookie 15479422554009591777 [ 4837.591476] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4837.592793] LustreError: 366053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4837.749637] Lustre: server umount lustre-MDT0001 complete [ 4842.745171] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 4843.878213] Key type lgssc unregistered [ 4844.002448] LNet: 368849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4844.004967] LNetError: 368849:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4844.016371] LNet: Removed LNI 192.168.204.136@tcp [ 4844.319265] Key type .llcrypt unregistered [ 4844.321456] Key type ._llcrypt unregistered [ 4854.688304] Key type ._llcrypt registered [ 4854.689622] Key type .llcrypt registered [ 4854.725458] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4854.980899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4855.004072] alg: No test for adler32 (adler32-zlib) [ 4855.870559] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 4855.959496] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 4857.559193] Key type lgssc registered [ 4858.152585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4861.051479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4861.060274] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4861.064823] LustreError: 370267:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 4861.075447] LustreError: 370267:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 4861.079271] LustreError: 370267:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 4861.082911] LustreError: 370267:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 4861.087492] LustreError: 370267:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4864.139311] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4864.142410] LustreError: 370663:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 4864.149792] LustreError: 370663:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 4864.151897] LustreError: 370663:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 4864.153981] LustreError: 370663:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 4864.156612] LustreError: 370663:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4870.369668] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 4871.324555] Key type lgssc unregistered [ 4871.454371] LNet: 371664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4871.456419] LNetError: 371664:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4871.464316] LNet: Removed LNI 192.168.204.136@tcp [ 4871.743460] Key type .llcrypt unregistered [ 4871.744635] Key type ._llcrypt unregistered [ 4877.562887] Key type ._llcrypt registered [ 4877.563895] Key type .llcrypt registered [ 4877.603981] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 4883.856841] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4884.184631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4884.229474] alg: No test for adler32 (adler32-zlib) [ 4885.072825] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 4885.149289] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 4886.727128] Key type lgssc registered [ 4887.086303] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4897.572979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4897.577841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4898.670307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4898.682994] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4898.720317] Lustre: lustre-MDT0000: new disk, initializing [ 4898.744211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4898.751748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4900.095182] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4904.274313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4904.314655] Lustre: 374793:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4904.335264] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4904.336872] Lustre: Skipped 1 previous similar message [ 4904.378825] Lustre: lustre-MDT0001: new disk, initializing [ 4904.414647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4904.426772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4904.431441] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4905.781872] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4909.622416] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4911.299791] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4913.558459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4913.681955] Lustre: lustre-OST0000: new disk, initializing [ 4913.684454] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4913.686962] Lustre: 376762:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4913.716900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4915.703942] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4917.756547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4917.760680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4917.800263] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4919.298047] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4920.636066] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4920.711141] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4922.057658] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4922.136532] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4927.969140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4927.969384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4927.971019] LustreError: Skipped 1 previous similar message [ 4927.971501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4927.973942] Lustre: Skipped 1 previous similar message [ 4927.977395] Lustre: Skipped 1 previous similar message [ 4933.093786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4933.098166] Lustre: Skipped 1 previous similar message [ 4937.183177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4937.262617] Lustre: server umount lustre-OST0000 complete [ 4940.256735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4940.261899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4940.271137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4943.328189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4943.328842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4948.448106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4948.449865] Lustre: Skipped 2 previous similar messages [ 4953.055074] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4953.117551] Lustre: server umount lustre-MDT0000 complete [ 4954.355710] LustreError: 374786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781665791 with bad export cookie 1313271504953860155 [ 4954.360464] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4954.481303] Lustre: server umount lustre-MDT0001 complete [ 4957.107483] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 4957.626737] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 23:09:54 (1781665794) [ 4959.727883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4959.896812] LustreError: 379221:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4959.905087] LustreError: 379221:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4959.923456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4961.261076] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4964.101832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4964.232575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4965.561586] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4967.853260] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4969.721643] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4972.088423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4974.228607] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4976.573468] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4978.829924] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4978.892874] Lustre: lustre-OST0001: new disk, initializing [ 4978.895379] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4978.898506] Lustre: 382053:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4978.933444] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4978.936301] Lustre: Skipped 1 previous similar message [ 4980.760305] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4984.297303] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4984.831689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4984.837605] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4984.866650] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4990.948625] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4990.950097] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4990.952615] Lustre: Skipped 1 previous similar message [ 4996.282878] Lustre: server umount lustre-OST0001 complete [ 5001.186554] LustreError: 381039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5001.191500] LustreError: 381039:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5006.307249] LustreError: 381041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5006.312283] LustreError: 381041:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5010.400247] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5010.403915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5010.410446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5010.412414] Lustre: Skipped 4 previous similar messages [ 5011.426818] LustreError: 381039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5011.430870] LustreError: 381039:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5014.281672] Lustre: server umount lustre-OST0000 complete [ 5015.519779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5015.522271] LustreError: Skipped 1 previous similar message [ 5021.652589] Lustre: server umount lustre-MDT0000 complete [ 5021.664123] LustreError: 379222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5021.669783] LustreError: 379222:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5022.807145] LustreError: 382052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781665860 with bad export cookie 1313271504953861023 [ 5022.810096] LustreError: 382052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5022.811118] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5028.970850] Lustre: server umount lustre-MDT0001 complete [ 5032.524610] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 5033.501796] Key type lgssc unregistered [ 5033.622316] LNet: 384034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5033.624581] LNetError: 384034:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5033.633259] LNet: Removed LNI 192.168.204.136@tcp [ 5033.924186] Key type .llcrypt unregistered [ 5033.925315] Key type ._llcrypt unregistered [ 5050.603023] Key type ._llcrypt registered [ 5050.603939] Key type .llcrypt registered [ 5050.639462] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 5051.071537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5051.085661] alg: No test for adler32 (adler32-zlib) [ 5051.932983] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 5052.016285] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 5053.599126] Key type lgssc registered [ 5053.942854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5056.357403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5056.364676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5057.455047] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5057.462454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5057.519352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5058.795912] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5061.450655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5061.483614] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5061.492896] Lustre: 386809:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5061.581854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5062.830222] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5065.063846] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5066.718920] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5071.839914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5071.842864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5071.846804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5072.352248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5072.352734] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5073.491756] Lustre: server umount lustre-MDT0000 complete [ 5074.651195] LustreError: 387548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781665911 with bad export cookie 16601039617650836076 [ 5074.654145] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5074.656438] LustreError: 387548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5074.761763] Lustre: server umount lustre-MDT0001 complete [ 5077.725493] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 23:11:54 (1781665914) [ 5079.749514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5079.914070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5081.197389] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5083.855941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5083.974738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5085.228918] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5087.519581] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5089.330840] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5091.481455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5091.548376] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5091.612454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5093.424731] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5095.597603] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5099.272593] Lustre: Permanently deactivating lustre-OST0000 [ 5099.277342] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5105.135842] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5106.459459] LustreError: 392349:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5106.463158] LustreError: 392349:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 5106.465638] LustreError: 392349:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5107.172220] LustreError: 392397:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 5107.175110] LustreError: 392397:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5114.028952] Lustre: server umount lustre-OST0000 complete [ 5115.351658] LustreError: 392646:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 5115.356055] LustreError: 392646:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5116.385554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5116.393433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5116.396974] Lustre: Skipped 2 previous similar messages [ 5121.504936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5121.508063] Lustre: Skipped 1 previous similar message [ 5126.626346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5126.633321] Lustre: Skipped 1 previous similar message [ 5126.680811] Lustre: server umount lustre-MDT0000 complete [ 5128.632855] LustreError: 389300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781665965 with bad export cookie 16601039617650836643 [ 5128.638658] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5128.640667] LustreError: 389300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5138.010578] Lustre: server umount lustre-MDT0001 complete [ 5142.895183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5144.962963] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5146.331817] LustreError: 393804:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5149.463911] Lustre: 394142:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.136@tcp [ 5150.453838] Lustre: server umount MGS complete [ 5155.921767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5156.169284] LustreError: 394866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5156.175411] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5156.177686] LustreError: 394866:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5156.181098] Lustre: Skipped 1 previous similar message [ 5156.221599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5158.192341] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5161.446068] LustreError: 394867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5166.581873] LustreError: 394866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5166.724284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5166.798119] hrtimer: interrupt took 4094902 ns [ 5167.629204] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5176.959385] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5191.381873] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5202.278576] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5204.861045] Lustre: Permanently reactivating lustre-OST0000 [ 5204.921891] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5204.935399] Lustre: Skipped 2 previous similar messages [ 5231.594511] LustreError: 396553:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5242.311843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5243.138388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5243.147663] Lustre: Skipped 1 previous similar message [ 5251.451648] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5260.926819] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5266.936151] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5266.936744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5266.976237] Lustre: Skipped 1 previous similar message [ 5266.980780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5266.993073] Lustre: Skipped 1 previous similar message [ 5281.247182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5281.426682] Lustre: server umount lustre-OST0000 complete [ 5285.348350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5285.359373] LustreError: Skipped 1 previous similar message [ 5285.371642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5285.393392] Lustre: Skipped 1 previous similar message [ 5285.412932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5285.415846] Lustre: Skipped 4 previous similar messages [ 5299.167132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5299.318778] Lustre: server umount lustre-MDT0000 complete [ 5302.759330] LustreError: 394867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5302.782374] LustreError: 394867:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5302.849383] LustreError: 396443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781666140 with bad export cookie 16601039617650837679 [ 5302.857303] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5302.860821] LustreError: 396443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5303.325811] Lustre: server umount lustre-MDT0001 complete [ 5313.324239] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 5316.044134] Key type lgssc unregistered [ 5316.370426] LNet: 398947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5316.379966] LNetError: 398947:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5316.408462] LNet: Removed LNI 192.168.204.136@tcp [ 5317.200852] Key type .llcrypt unregistered [ 5317.204409] Key type ._llcrypt unregistered [ 5335.188119] Key type ._llcrypt registered [ 5335.190469] Key type .llcrypt registered [ 5335.314135] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 5351.523364] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 5352.550269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5352.666794] alg: No test for adler32 (adler32-zlib) [ 5353.778160] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 5353.960529] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 5355.615309] Key type lgssc registered [ 5356.947604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5390.727565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5390.761548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5392.049710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5392.079604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5392.180331] Lustre: lustre-MDT0000: new disk, initializing [ 5392.281461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5392.305349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5396.601022] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5406.504330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5406.628253] Lustre: 402110:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5406.697356] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5406.702393] Lustre: Skipped 1 previous similar message [ 5406.795230] Lustre: lustre-MDT0001: new disk, initializing [ 5406.908518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5406.957489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5406.968229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5411.413885] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5421.414992] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5427.229599] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5434.738538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5435.103311] Lustre: lustre-OST0000: new disk, initializing [ 5435.107407] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5435.116200] Lustre: 404077:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5435.215658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5442.145824] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5443.164261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5443.177717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5443.275811] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5451.666527] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5457.068373] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5457.362102] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5462.447267] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5462.653506] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5469.154382] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5469.155788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5469.169129] LustreError: Skipped 1 previous similar message [ 5469.172082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5469.190907] Lustre: Skipped 1 previous similar message [ 5469.216767] Lustre: Skipped 1 previous similar message [ 5474.284729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5474.297444] Lustre: Skipped 1 previous similar message [ 5479.394746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5479.403241] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 5479.408124] Lustre: Skipped 1 previous similar message [ 5479.590427] Lustre: server umount lustre-OST0000 complete [ 5484.512639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5484.529271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5489.124884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5489.143174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5489.173527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5490.535335] Lustre: server umount lustre-MDT0000 complete [ 5494.709994] LustreError: 402102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781666332 with bad export cookie 9869092632834690014 [ 5494.717863] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5494.733096] LustreError: 402102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5495.449756] Lustre: server umount lustre-MDT0001 complete [ 5504.947268] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5507.340840] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 23:19:02 (1781666342) [ 5516.047826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5516.974972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5522.131086] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5522.412488] LustreError: 406541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5522.437776] LustreError: 406541:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5527.520918] LustreError: 406540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5530.435804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5530.747903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5534.569673] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5541.540507] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5549.078405] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5561.403849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5562.233770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5572.463211] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5583.395368] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5586.253430] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5589.064777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5596.128670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5596.131444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5596.144093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5596.144114] Lustre: Skipped 1 previous similar message [ 5596.186501] Lustre: Skipped 1 previous similar message [ 5601.262554] LustreError: 408365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5601.287153] LustreError: 408365:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5601.509773] Lustre: server umount lustre-OST0000 complete [ 5606.377564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5606.393521] Lustre: Skipped 1 previous similar message [ 5612.593332] Lustre: server umount lustre-MDT0000 complete [ 5616.536400] LustreError: 408385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781666453 with bad export cookie 9869092632834690896 [ 5616.539558] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5616.545304] LustreError: 408385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5616.850838] Lustre: server umount lustre-MDT0001 complete [ 5625.607433] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 5627.844138] Key type lgssc unregistered [ 5628.059302] LNet: 410268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5628.065460] LNetError: 410268:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5628.079395] LNet: Removed LNI 192.168.204.136@tcp [ 5628.694705] Key type .llcrypt unregistered [ 5628.696371] Key type ._llcrypt unregistered [ 5646.015376] Key type ._llcrypt registered [ 5646.018537] Key type .llcrypt registered [ 5646.116619] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 5646.353910] Key type .llcrypt unregistered [ 5646.358412] Key type ._llcrypt unregistered [ 5659.168834] Key type ._llcrypt registered [ 5659.170926] Key type .llcrypt registered [ 5659.267076] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 5660.118983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5660.181581] alg: No test for adler32 (adler32-zlib) [ 5661.146941] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 5661.353203] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 5662.999451] Key type lgssc registered [ 5663.896647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5669.716534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5669.733220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5671.100302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5673.957095] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5678.837532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5679.001050] LustreError: 412460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5679.009868] LustreError: 412460:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5679.024399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5681.061421] LustreError: 413205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5681.069308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5681.071932] LustreError: 413205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5682.296212] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5686.033313] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5686.243854] LustreError: 412461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5686.251923] LustreError: 412461:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5689.434423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5689.634107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5691.681676] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5701.600940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5701.609984] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5701.631436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5705.187809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5705.188307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5705.192407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5706.871672] Lustre: server umount lustre-OST0000 complete [ 5709.798641] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5709.803172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5709.814202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5709.817834] Lustre: Skipped 1 previous similar message [ 5711.290902] Lustre: server umount lustre-MDT0000 complete [ 5713.567648] LustreError: 414200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781666550 with bad export cookie 10312623996063809677 [ 5713.571698] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5713.575066] LustreError: 414200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5713.846316] Lustre: server umount lustre-MDT0001 complete [ 5721.216753] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 5723.010960] Key type lgssc unregistered [ 5723.221674] LNet: 415604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5723.225868] LNetError: 415604:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5723.236775] LNet: Removed LNI 192.168.204.136@tcp [ 5723.754681] Key type .llcrypt unregistered [ 5723.756169] Key type ._llcrypt unregistered [ 5737.645618] Key type ._llcrypt registered [ 5737.647254] Key type .llcrypt registered [ 5737.709284] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 5738.148576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5738.162153] alg: No test for adler32 (adler32-zlib) [ 5739.049615] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 5739.164556] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 5740.775116] Key type lgssc registered [ 5741.320915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5745.892488] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5745.901045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5761.375455] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5766.495222] LustreError: 416971:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.136@tcp: failed processing log, type 4: rc = -110 [ 5793.120421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5796.680144] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5800.859692] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5804.624706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5804.913076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5804.969088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5807.475805] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5812.457478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5812.664069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5814.918129] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5819.036527] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5822.560405] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5828.071809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5828.073444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5828.090383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5828.097677] Lustre: Skipped 1 previous similar message [ 5833.186645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5833.192700] Lustre: Skipped 1 previous similar message [ 5834.052505] Lustre: server umount lustre-OST0000 complete [ 5836.497881] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5836.500923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5836.518190] Lustre: Skipped 1 previous similar message [ 5836.527190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5838.306386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5842.677455] Lustre: server umount lustre-MDT0000 complete [ 5843.424716] LustreError: 418007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.441164] LustreError: 418007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5844.918469] LustreError: 416979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781666682 with bad export cookie 9892084621569338390 [ 5844.921993] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5844.924899] LustreError: 416979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5845.112363] Lustre: server umount lustre-MDT0001 complete [ 5851.161134] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 5852.778546] Key type lgssc unregistered [ 5852.960928] LNet: 420539:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5852.964676] LNetError: 420539:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5852.979031] LNet: Removed LNI 192.168.204.136@tcp [ 5853.426411] Key type .llcrypt unregistered [ 5853.428252] Key type ._llcrypt unregistered [ 5866.822141] Key type ._llcrypt registered [ 5866.823415] Key type .llcrypt registered [ 5866.878823] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 5867.314382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5867.327471] alg: No test for adler32 (adler32-zlib) [ 5868.272795] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 5868.414493] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 5870.039516] Key type lgssc registered [ 5870.720358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5875.362717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5875.376247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5876.688260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5879.010746] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5883.890382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5884.112897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5886.348176] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5890.212634] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5893.076409] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5896.505371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5896.690574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5899.363663] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5900.777101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5902.931078] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5909.281296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5909.284044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5909.299850] Lustre: Skipped 2 previous similar messages [ 5909.303316] Lustre: Skipped 1 previous similar message [ 5910.640331] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.36@tcp (stopping) [ 5912.354928] Lustre: server umount lustre-MDT0000 complete [ 5913.273090] LustreError: 421914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.36@tcp arrived at 1781666750 with bad export cookie 2581237805915305454 [ 5914.202516] LustreError: 421915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781666751 with bad export cookie 2581237805915304579 [ 5914.208586] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5914.209433] LustreError: 421915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5914.591984] LustreError: 421933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5914.592398] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5914.601353] LustreError: 421933:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5914.608719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5916.339669] Lustre: server umount lustre-OST0000 complete [ 5919.712331] LustreError: 422662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5919.721411] LustreError: 422662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5921.567259] LustreError: 422664:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000001041b65 x1868212946887552/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5921.660455] Lustre: server umount lustre-MDT0001 complete [ 5928.640749] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 5930.946989] Key type lgssc unregistered [ 5931.221128] LNet: 425614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5931.228304] LNetError: 425614:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5931.240202] LNet: Removed LNI 192.168.204.136@tcp [ 5931.986171] Key type .llcrypt unregistered [ 5931.988170] Key type ._llcrypt unregistered [ 5952.025426] Key type ._llcrypt registered [ 5952.028220] Key type .llcrypt registered [ 5952.126561] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 5953.102302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5953.193551] alg: No test for adler32 (adler32-zlib) [ 5954.285819] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 5954.550308] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 5956.271133] Key type lgssc registered [ 5957.221779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5964.017424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5964.029795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5965.419942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5968.035812] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5972.649655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5972.840064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5974.861904] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5978.511876] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5981.164579] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5984.513894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5984.719624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5987.844878] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5989.869762] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5991.551174] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5993.709382] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5993.715887] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5993.718689] Lustre: Skipped 1 previous similar message [ 5998.184588] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.36@tcp (stopping) [ 6003.306841] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.36@tcp (stopping) [ 6003.312231] Lustre: Skipped 2 previous similar messages [ 6008.287134] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6008.350347] Lustre: server umount lustre-MDT0001 complete [ 6008.425036] LustreError: 427996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6008.432404] LustreError: 427996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6009.312835] LustreError: 427011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6009.322792] LustreError: 427011:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6014.432694] LustreError: 427738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6014.434047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6014.446464] Lustre: Skipped 1 previous similar message [ 6014.448736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6014.451461] Lustre: Skipped 2 previous similar messages [ 6016.466904] Lustre: server umount lustre-OST0000 complete [ 6018.041865] Lustre: server umount lustre-MDT0000 complete [ 6023.026972] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 6024.195878] Key type lgssc unregistered [ 6024.338885] LNet: 430695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6024.343484] LNetError: 430695:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6024.356372] LNet: Removed LNI 192.168.204.136@tcp [ 6024.676329] Key type .llcrypt unregistered [ 6024.678144] Key type ._llcrypt unregistered [ 6033.277733] Key type ._llcrypt registered [ 6033.278907] Key type .llcrypt registered [ 6033.319169] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 6039.654241] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 6039.950965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6039.967535] alg: No test for adler32 (adler32-zlib) [ 6040.832767] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 6040.937885] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 6042.519095] Key type lgssc registered [ 6042.905810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6055.849308] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 6061.793728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6061.801771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6062.897518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6062.909727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6062.946482] Lustre: lustre-MDT0000: new disk, initializing [ 6062.979284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6062.987399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6064.387049] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6070.296485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6070.399927] Lustre: lustre-OST0000: new disk, initializing [ 6070.402100] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6070.403793] Lustre: Skipped 1 previous similar message [ 6070.405924] Lustre: 435699:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6070.432511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6071.106081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6071.110188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6071.121209] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6072.529381] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6076.443099] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6082.003190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6082.050712] Lustre: 434679:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6082.079962] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6082.117255] Lustre: lustre-MDT0001: new disk, initializing [ 6082.164864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6082.180620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6082.184905] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6083.775559] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6087.644311] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6087.736272] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6089.334542] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6090.449558] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [ 6092.915720] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.36@tcp (stopping) [ 6095.329346] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6095.332312] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6095.337779] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6095.340855] Lustre: Skipped 1 previous similar message [ 6098.026641] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.36@tcp (stopping) [ 6098.517185] Lustre: server umount lustre-MDT0001 complete [ 6105.568785] LustreError: 434689:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6105.574113] LustreError: 434689:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6110.690491] LustreError: 434687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6110.695733] LustreError: 434687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6112.224087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6112.226877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6112.230919] Lustre: Skipped 1 previous similar message [ 6112.233074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6115.809019] LustreError: 434692:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6117.073785] Lustre: server umount lustre-OST0000 complete [ 6124.541493] Lustre: server umount lustre-MDT0000 complete [ 6128.982625] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 6130.024297] Key type lgssc unregistered [ 6130.150428] LNet: 439451:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6130.153764] LNetError: 439451:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6130.162436] LNet: Removed LNI 192.168.204.136@tcp [ 6130.479290] Key type .llcrypt unregistered [ 6130.480323] Key type ._llcrypt unregistered [ 6143.201413] Key type ._llcrypt registered [ 6143.202560] Key type .llcrypt registered [ 6143.235634] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 6143.326355] Key type .llcrypt unregistered [ 6143.327357] Key type ._llcrypt unregistered [ 6149.874333] Key type ._llcrypt registered [ 6149.875653] Key type .llcrypt registered [ 6149.911991] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 6150.213653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6150.223512] alg: No test for adler32 (adler32-zlib) [ 6151.076904] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 6151.156490] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 6152.735162] Key type lgssc registered [ 6153.075604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6161.308473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6161.314287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6162.417706] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6162.430460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6162.467414] Lustre: lustre-MDT0000: new disk, initializing [ 6162.493882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6162.501153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6163.864883] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6167.956634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6168.005463] Lustre: 442799:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6168.029716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6168.032555] Lustre: Skipped 1 previous similar message [ 6168.083642] Lustre: lustre-MDT0001: new disk, initializing [ 6168.115600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6168.126658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6168.131113] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6169.477190] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6173.264902] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6174.975763] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6177.314296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6177.433890] Lustre: lustre-OST0000: new disk, initializing [ 6177.436837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6177.441462] Lustre: 444766:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6177.479426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6179.501215] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6180.599432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6180.602801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6180.633451] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6183.299588] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6185.953415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6185.954518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6185.957084] Lustre: Skipped 1 previous similar message [ 6185.960820] Lustre: Skipped 2 previous similar messages [ 6191.072846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6191.076926] Lustre: Skipped 2 previous similar messages [ 6191.832544] Lustre: server umount lustre-MDT0000 complete [ 6193.082328] LustreError: 442791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781667030 with bad export cookie 4691569361258571779 [ 6193.086114] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6193.088230] LustreError: 442791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6193.201588] Lustre: server umount lustre-MDT0001 complete [ 6204.710232] Lustre: server umount lustre-OST0000 complete [ 6208.209907] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 23:30:44 (1781667044) [ 6210.637109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6210.827588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6212.048794] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6214.636140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6215.934538] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6218.053968] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6219.744702] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6234.913868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6234.979906] Lustre: Found index 0 for lustre-OST0000, updating log [ 6235.046765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6235.048931] Lustre: Skipped 1 previous similar message [ 6236.070406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6236.953759] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6239.358717] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6241.247800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6241.250145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6241.253806] Lustre: Skipped 1 previous similar message [ 6241.255484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6245.346268] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6245.348309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6245.348940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6246.674594] Lustre: server umount lustre-OST0000 complete [ 6259.362890] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6259.363713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6259.366621] Lustre: Skipped 1 previous similar message [ 6259.368779] Lustre: Skipped 1 previous similar message [ 6265.503764] Lustre: server umount lustre-MDT0000 complete [ 6265.824809] LustreError: 446791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6265.833696] LustreError: 446791:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6266.803268] LustreError: 446771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781667104 with bad export cookie 4691569361258573158 [ 6266.806483] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6266.808822] LustreError: 446771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6266.937164] Lustre: server umount lustre-MDT0001 complete [ 6269.857844] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 23:31:46 (1781667106) [ 6300.639197] LNet: 1 local NIs in recovery (showing 1): 192.168.204.136@tcp [ 6849.124752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6849.279079] LustreError: 453190:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6849.301239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6850.632217] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6853.407186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6854.903509] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6857.275889] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6858.954177] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6874.084965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6874.163215] Lustre: Found index 0 for lustre-OST0000, updating log [ 6874.267946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6874.270603] Lustre: Skipped 1 previous similar message [ 6876.311651] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6877.350489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 6878.519736] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6880.162094] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6880.166990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6880.168832] Lustre: Skipped 3 previous similar messages [ 6881.185231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6885.255219] Lustre: server umount lustre-OST0000 complete [ 6887.392506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6887.393494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6887.397529] Lustre: Skipped 1 previous similar message [ 6891.487995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6891.489824] Lustre: Skipped 8 previous similar messages [ 6892.498889] Lustre: server umount lustre-MDT0000 complete [ 6892.512233] LustreError: 453191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6892.517597] LustreError: 453191:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6893.660935] LustreError: 453172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781667730 with bad export cookie 4691569361258574222 [ 6893.664029] LustreError: 453172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6893.664109] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6893.774273] Lustre: server umount lustre-MDT0001 complete [ 6901.727353] LNet: 1 local NIs in recovery (showing 1): 192.168.204.136@tcp [ 6910.296754] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 6912.967934] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 6922.964335] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 23:42:39 (1781667759) [ 6928.024343] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 6929.024178] Key type lgssc unregistered [ 6937.375687] LNet: 459831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6937.378046] LNetError: 459831:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7005.989358] LNet: Removed LNI 192.168.204.136@tcp [ 7006.281254] Key type .llcrypt unregistered [ 7006.283030] Key type ._llcrypt unregistered [ 7012.786716] Key type ._llcrypt registered [ 7012.787803] Key type .llcrypt registered [ 7012.823905] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 7013.174083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7013.210531] alg: No test for adler32 (adler32-zlib) [ 7014.065311] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 7014.143044] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 7015.719141] Key type lgssc registered [ 7016.074932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7017.488181] LNet: Added LNI 192.168.204.136@tcp100 [8/256/0/180] [ 7017.785884] LNet: Added LNI 192.168.204.136@tcp101 [8/256/0/180] [ 7018.072300] LNet: Added LNI 192.168.204.136@tcp102 [8/256/0/180] [ 7018.361573] LNet: Added LNI 192.168.204.136@tcp103 [8/256/0/180] [ 7018.653876] LNet: Added LNI 192.168.204.136@tcp104 [8/256/0/180] [ 7018.950800] LNet: Added LNI 192.168.204.136@tcp105 [8/256/0/180] [ 7019.233669] LNet: Added LNI 192.168.204.136@tcp106 [8/256/0/180] [ 7019.521986] LNet: Added LNI 192.168.204.136@tcp107 [8/256/0/180] [ 7019.806694] LNet: Added LNI 192.168.204.136@tcp108 [8/256/0/180] [ 7020.097937] LNet: Added LNI 192.168.204.136@tcp109 [8/256/0/180] [ 7020.375251] LNet: Added LNI 192.168.204.136@tcp110 [8/256/0/180] [ 7020.639480] LNet: Added LNI 192.168.204.136@tcp111 [8/256/0/180] [ 7020.911301] LNet: Added LNI 192.168.204.136@tcp112 [8/256/0/180] [ 7021.200895] LNet: Added LNI 192.168.204.136@tcp113 [8/256/0/180] [ 7021.489664] LNet: Added LNI 192.168.204.136@tcp114 [8/256/0/180] [ 7021.781593] LNet: Added LNI 192.168.204.136@tcp115 [8/256/0/180] [ 7022.064626] LNet: Added LNI 192.168.204.136@tcp116 [8/256/0/180] [ 7022.351459] LNet: Added LNI 192.168.204.136@tcp117 [8/256/0/180] [ 7022.634445] LNet: Added LNI 192.168.204.136@tcp118 [8/256/0/180] [ 7022.927431] LNet: Added LNI 192.168.204.136@tcp119 [8/256/0/180] [ 7023.225819] LNet: Added LNI 192.168.204.136@tcp120 [8/256/0/180] [ 7023.516149] LNet: Added LNI 192.168.204.136@tcp121 [8/256/0/180] [ 7023.828613] LNet: Added LNI 192.168.204.136@tcp122 [8/256/0/180] [ 7024.140652] LNet: Added LNI 192.168.204.136@tcp123 [8/256/0/180] [ 7024.445531] LNet: Added LNI 192.168.204.136@tcp124 [8/256/0/180] [ 7024.758888] LNet: Added LNI 192.168.204.136@tcp125 [8/256/0/180] [ 7025.078527] LNet: Added LNI 192.168.204.136@tcp126 [8/256/0/180] [ 7025.400413] LNet: Added LNI 192.168.204.136@tcp127 [8/256/0/180] [ 7025.708188] LNet: Added LNI 192.168.204.136@tcp128 [8/256/0/180] [ 7025.997366] LNet: Added LNI 192.168.204.136@tcp129 [8/256/0/180] [ 7026.285789] LNet: Added LNI 192.168.204.136@tcp130 [8/256/0/180] [ 7026.584101] LNet: Added LNI 192.168.204.136@tcp131 [8/256/0/180] [ 7026.872966] LNet: Added LNI 192.168.204.136@tcp132 [8/256/0/180] [ 7027.160811] LNet: Added LNI 192.168.204.136@tcp133 [8/256/0/180] [ 7027.453602] LNet: Added LNI 192.168.204.136@tcp134 [8/256/0/180] [ 7027.750690] LNet: Added LNI 192.168.204.136@tcp135 [8/256/0/180] [ 7029.995836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7030.001906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7031.062252] Lustre: 462829:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.136@tcp: can't add NID 192.168.204.136@tcp130: rc = -75 [ 7031.135504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7031.150952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7031.196420] Lustre: lustre-MDT0000: new disk, initializing [ 7031.227983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7031.234863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7032.100954] LNetError: 460292:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.136 [ 7032.105748] LNetError: Protocol error connecting to 192.168.204.136@tcp110 at host 192.168.204.136:988: is it running a compatible version of Lustre? [ 7032.645904] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7036.768848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7036.823447] Lustre: 462845:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7036.851813] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7036.853654] Lustre: Skipped 1 previous similar message [ 7036.894944] Lustre: lustre-MDT0001: new disk, initializing [ 7036.908350] Lustre: 463647:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.136@tcp: can't add NID 192.168.204.136@tcp130: rc = -75 [ 7036.910979] Lustre: 463647:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7036.958307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7036.971802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7036.976807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7038.286644] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7039.926756] Lustre: 464020:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.136@tcp: can't add NID 192.168.204.136@tcp130: rc = -75 [ 7039.929903] Lustre: 464020:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7041.895555] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7043.638944] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7045.874371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7045.984111] Lustre: lustre-OST0000: new disk, initializing [ 7045.985664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7045.989289] Lustre: 464819:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7046.027168] Lustre: 464824:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.136@tcp: can't add NID 192.168.204.136@tcp130: rc = -75 [ 7046.030186] Lustre: 464824:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 7046.046672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7047.907746] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7048.261376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7048.264227] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7048.316853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7051.562851] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7053.795506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7053.797976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7053.802619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7058.399818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7058.402413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7058.408426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7064.033873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7064.035885] Lustre: Skipped 2 previous similar messages [ 7068.127093] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7068.174589] Lustre: server umount lustre-OST0000 complete [ 7072.544032] LNetError: 460295:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.136 [ 7072.547113] LNetError: Protocol error connecting to 192.168.204.136@tcp110 at host 192.168.204.136:988: is it running a compatible version of Lustre? [ 7072.735756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7072.738654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7072.743715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7075.476521] Lustre: server umount lustre-MDT0000 complete [ 7076.709071] LustreError: 462839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781667914 with bad export cookie 8081010498458909313 [ 7076.711724] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7076.715582] LustreError: 462839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7076.836544] Lustre: server umount lustre-MDT0001 complete [ 7080.403145] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 7081.519673] Key type lgssc unregistered [ 7081.679792] LNet: 466730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7081.682687] LNetError: 466730:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7081.684894] LNet: Removed LNI 192.168.204.136@tcp [ 7082.719178] LNet: Removed LNI 192.168.204.136@tcp100 [ 7082.720425] LNetError: 466730:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7082.722692] LNetError: 466730:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7082.725357] LNet: Removed LNI 192.168.204.136@tcp101 [ 7083.743139] LNet: Removed LNI 192.168.204.136@tcp102 [ 7083.745290] LNetError: 466730:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7083.748316] LNetError: 466730:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7083.750376] LNet: Removed LNI 192.168.204.136@tcp103 [ 7084.767121] LNet: Removed LNI 192.168.204.136@tcp104 [ 7085.791109] LNet: Removed LNI 192.168.204.136@tcp105 [ 7085.792383] LNetError: 466730:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7085.794616] LNetError: 466730:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7085.796904] LNet: Removed LNI 192.168.204.136@tcp106 [ 7086.815117] LNet: Removed LNI 192.168.204.136@tcp107 [ 7087.839142] LNet: Removed LNI 192.168.204.136@tcp108 [ 7088.863164] LNet: Removed LNI 192.168.204.136@tcp109 [ 7089.887091] LNet: Removed LNI 192.168.204.136@tcp110 [ 7089.888574] LNetError: 466730:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7089.890945] LNetError: 466730:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7089.893665] LNet: Removed LNI 192.168.204.136@tcp111 [ 7090.911129] LNet: Removed LNI 192.168.204.136@tcp112 [ 7091.935148] LNet: Removed LNI 192.168.204.136@tcp113 [ 7092.959095] LNet: Removed LNI 192.168.204.136@tcp114 [ 7093.983137] LNet: Removed LNI 192.168.204.136@tcp115 [ 7095.007099] LNet: Removed LNI 192.168.204.136@tcp116 [ 7095.008918] LNet: Removed LNI 192.168.204.136@tcp117 [ 7095.010157] LNet: Removed LNI 192.168.204.136@tcp118 [ 7095.011192] LNet: Removed LNI 192.168.204.136@tcp119 [ 7095.012186] LNet: Removed LNI 192.168.204.136@tcp120 [ 7095.013161] LNet: Removed LNI 192.168.204.136@tcp121 [ 7095.014163] LNet: Removed LNI 192.168.204.136@tcp122 [ 7095.015174] LNet: Removed LNI 192.168.204.136@tcp123 [ 7095.016229] LNet: Removed LNI 192.168.204.136@tcp124 [ 7095.017268] LNet: Removed LNI 192.168.204.136@tcp125 [ 7095.018336] LNet: Removed LNI 192.168.204.136@tcp126 [ 7095.019401] LNet: Removed LNI 192.168.204.136@tcp127 [ 7095.020468] LNet: Removed LNI 192.168.204.136@tcp128 [ 7095.021531] LNet: Removed LNI 192.168.204.136@tcp129 [ 7095.022525] LNet: Removed LNI 192.168.204.136@tcp130 [ 7095.023615] LNet: Removed LNI 192.168.204.136@tcp131 [ 7095.024688] LNet: Removed LNI 192.168.204.136@tcp132 [ 7095.025722] LNet: Removed LNI 192.168.204.136@tcp133 [ 7095.026701] LNet: Removed LNI 192.168.204.136@tcp134 [ 7095.033446] LNet: Removed LNI 192.168.204.136@tcp135 [ 7095.296233] Key type .llcrypt unregistered [ 7095.298790] Key type ._llcrypt unregistered [ 7102.461814] Key type ._llcrypt registered [ 7102.463268] Key type .llcrypt registered [ 7102.500074] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 7108.487255] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 7108.896333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7108.926136] alg: No test for adler32 (adler32-zlib) [ 7109.773841] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 7109.859053] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 7111.439191] Key type lgssc registered [ 7111.794752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7122.769437] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 23:45:59 (1781667959) [ 7128.447512] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 7129.478479] Key type lgssc unregistered [ 7129.596304] LNet: 470789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7129.598620] LNetError: 470789:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7129.605338] LNet: Removed LNI 192.168.204.136@tcp [ 7129.898397] Key type .llcrypt unregistered [ 7129.899602] Key type ._llcrypt unregistered [ 7136.446566] Key type ._llcrypt registered [ 7136.447772] Key type .llcrypt registered [ 7136.487070] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 7136.912596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7136.933977] alg: No test for adler32 (adler32-zlib) [ 7137.796597] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 7137.881790] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 7139.463192] Key type lgssc registered [ 7139.820463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7142.837597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7142.844877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7143.941473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7143.952929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7143.992686] Lustre: lustre-MDT0000: new disk, initializing [ 7144.016964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7144.025753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7145.390743] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7149.362652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7149.401782] Lustre: 471976:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7149.428825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7149.430972] Lustre: Skipped 1 previous similar message [ 7149.466323] Lustre: lustre-MDT0001: new disk, initializing [ 7149.498680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7149.509831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7149.512878] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7150.835771] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7154.413823] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7156.125991] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7158.381566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7158.505239] Lustre: lustre-OST0000: new disk, initializing [ 7158.507184] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7158.510292] Lustre: 473947:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7158.537655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7159.547644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7159.553283] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7159.585041] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7160.594747] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7164.341106] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7165.580946] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing lnet_if_list [ 7167.253638] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7167.266563] LNet: Added LNI 192.168.204.136@tcp10 [8/256/0/180] [ 7169.088149] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7169.100722] LNet: Added LNI 192.168.204.136@tcp11 [8/256/0/180] [ 7170.465291] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7170.478191] LNet: Added LNI 192.168.204.136@tcp12 [8/256/0/180] [ 7171.818681] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7171.829408] LNet: Added LNI 192.168.204.136@tcp13 [8/256/0/180] [ 7173.170221] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7173.181088] LNet: Added LNI 192.168.204.136@tcp14 [8/256/0/180] [ 7174.549075] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7174.559659] LNet: Added LNI 192.168.204.136@tcp15 [8/256/0/180] [ 7175.986318] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7175.995857] LNet: Added LNI 192.168.204.136@tcp16 [8/256/0/180] [ 7177.449321] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7177.459291] LNet: Added LNI 192.168.204.136@tcp17 [8/256/0/180] [ 7178.938173] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7178.948196] LNet: Added LNI 192.168.204.136@tcp18 [8/256/0/180] [ 7180.436269] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7180.448196] LNet: Added LNI 192.168.204.136@tcp19 [8/256/0/180] [ 7182.016224] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7182.029679] LNet: Added LNI 192.168.204.136@tcp20 [8/256/0/180] [ 7183.726042] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7183.737853] LNet: Added LNI 192.168.204.136@tcp21 [8/256/0/180] [ 7185.494304] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7185.508420] LNet: Added LNI 192.168.204.136@tcp22 [8/256/0/180] [ 7187.225679] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7187.236622] LNet: Added LNI 192.168.204.136@tcp23 [8/256/0/180] [ 7188.972061] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7188.982958] LNet: Added LNI 192.168.204.136@tcp24 [8/256/0/180] [ 7190.730908] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7190.745812] LNet: Added LNI 192.168.204.136@tcp25 [8/256/0/180] [ 7192.480177] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7192.490561] LNet: Added LNI 192.168.204.136@tcp26 [8/256/0/180] [ 7194.275698] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7194.286631] LNet: Added LNI 192.168.204.136@tcp27 [8/256/0/180] [ 7196.139441] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7196.152052] LNet: Added LNI 192.168.204.136@tcp28 [8/256/0/180] [ 7197.998578] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7198.008913] LNet: Added LNI 192.168.204.136@tcp29 [8/256/0/180] [ 7200.002837] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7200.015355] LNet: Added LNI 192.168.204.136@tcp30 [8/256/0/180] [ 7202.089739] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7202.103428] LNet: Added LNI 192.168.204.136@tcp31 [8/256/0/180] [ 7204.404937] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7204.416379] LNet: Added LNI 192.168.204.136@tcp32 [8/256/0/180] [ 7206.694981] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7206.707289] LNet: Added LNI 192.168.204.136@tcp33 [8/256/0/180] [ 7208.770890] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7208.785213] LNet: Added LNI 192.168.204.136@tcp34 [8/256/0/180] [ 7210.941907] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7210.954444] LNet: Added LNI 192.168.204.136@tcp35 [8/256/0/180] [ 7213.011943] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7213.021899] LNet: Added LNI 192.168.204.136@tcp36 [8/256/0/180] [ 7215.193482] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7215.205935] LNet: Added LNI 192.168.204.136@tcp37 [8/256/0/180] [ 7217.507994] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7217.518813] LNet: Added LNI 192.168.204.136@tcp38 [8/256/0/180] [ 7219.820772] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7219.831718] LNet: Added LNI 192.168.204.136@tcp39 [8/256/0/180] [ 7222.090402] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7222.100448] LNet: Added LNI 192.168.204.136@tcp40 [8/256/0/180] [ 7224.349365] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7224.360701] LNet: Added LNI 192.168.204.136@tcp41 [8/256/0/180] [ 7226.640750] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7226.654053] LNet: Added LNI 192.168.204.136@tcp42 [8/256/0/180] [ 7229.058267] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7229.068297] LNet: Added LNI 192.168.204.136@tcp43 [8/256/0/180] [ 7231.517597] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7231.529596] LNet: Added LNI 192.168.204.136@tcp44 [8/256/0/180] [ 7234.034271] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7234.048567] LNet: Added LNI 192.168.204.136@tcp45 [8/256/0/180] [ 7236.637934] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7236.649049] LNet: Added LNI 192.168.204.136@tcp46 [8/256/0/180] [ 7239.294201] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7239.309423] LNet: Added LNI 192.168.204.136@tcp47 [8/256/0/180] [ 7241.901112] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7241.912088] LNet: Added LNI 192.168.204.136@tcp48 [8/256/0/180] [ 7244.523886] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7244.539314] LNet: Added LNI 192.168.204.136@tcp49 [8/256/0/180] [ 7247.423741] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7247.434846] LNet: Added LNI 192.168.204.136@tcp50 [8/256/0/180] [ 7257.056807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7257.059262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7257.067646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7257.069876] Lustre: Skipped 2 previous similar messages [ 7260.640056] Lustre: server umount lustre-MDT0000 complete [ 7261.797142] LustreError: 471968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781668099 with bad export cookie 8198792785047897191 [ 7261.800995] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7261.802479] LustreError: 471968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7261.910395] Lustre: server umount lustre-MDT0001 complete [ 7268.844716] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 7271.627790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7272.415851] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x71c7f69953d0cc78 [ 7272.420118] Lustre: MGC192.168.204.136@tcp: Connection restored to 0@lo (at 0@lo) [ 7272.537086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7273.775076] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7276.458099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7276.547599] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7276.553986] Lustre: Skipped 2 previous similar messages [ 7276.582131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7277.604910] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7277.892523] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7278.559098] Lustre: 471252:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781668099/real 1781668099] req@0000000044b4b173 x1868214278636160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781668115 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7280.327956] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7282.088554] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7283.679207] Lustre: 471253:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781668103/real 1781668103] req@00000000dde73d90 x1868214278636288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781668119 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7283.688202] Lustre: 471253:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7288.248098] Lustre: 471252:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781668109/real 1781668109] req@00000000dab04fd3 x1868214278636544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781668125 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7291.872271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7291.874761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7291.879371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7292.897566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7292.899815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7292.902983] Lustre: Skipped 1 previous similar message [ 7295.575175] Lustre: server umount lustre-OST0000 complete [ 7296.991883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7296.996114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7297.002095] Lustre: Skipped 1 previous similar message [ 7297.003879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7303.136262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7303.138222] Lustre: Skipped 3 previous similar messages [ 7311.327143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7311.390506] Lustre: server umount lustre-MDT0000 complete [ 7312.620284] LustreError: 471968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781668149 with bad export cookie 8198792785047899256 [ 7312.623438] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7312.625620] LustreError: 471968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7312.742448] Lustre: server umount lustre-MDT0001 complete [ 7316.257132] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 7317.274500] Key type lgssc unregistered [ 7317.423223] LNet: 500908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7317.426254] LNetError: 500908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7317.428577] LNet: Removed LNI 192.168.204.136@tcp [ 7317.429708] LNet: Removed LNI 192.168.204.136@tcp10 [ 7317.430897] LNet: Removed LNI 192.168.204.136@tcp11 [ 7317.432083] LNet: Removed LNI 192.168.204.136@tcp12 [ 7317.433220] LNet: Removed LNI 192.168.204.136@tcp13 [ 7317.434392] LNet: Removed LNI 192.168.204.136@tcp14 [ 7317.435544] LNet: Removed LNI 192.168.204.136@tcp15 [ 7317.436776] LNet: Removed LNI 192.168.204.136@tcp16 [ 7317.437916] LNet: Removed LNI 192.168.204.136@tcp17 [ 7317.439101] LNet: Removed LNI 192.168.204.136@tcp18 [ 7317.440227] LNet: Removed LNI 192.168.204.136@tcp19 [ 7317.441323] LNet: Removed LNI 192.168.204.136@tcp20 [ 7317.442454] LNet: Removed LNI 192.168.204.136@tcp21 [ 7317.443627] LNet: Removed LNI 192.168.204.136@tcp22 [ 7317.444781] LNet: Removed LNI 192.168.204.136@tcp23 [ 7317.445888] LNet: Removed LNI 192.168.204.136@tcp24 [ 7317.447028] LNet: Removed LNI 192.168.204.136@tcp25 [ 7317.448230] LNet: Removed LNI 192.168.204.136@tcp26 [ 7317.449396] LNet: Removed LNI 192.168.204.136@tcp27 [ 7317.450586] LNet: Removed LNI 192.168.204.136@tcp28 [ 7317.451798] LNet: Removed LNI 192.168.204.136@tcp29 [ 7317.452966] LNet: Removed LNI 192.168.204.136@tcp30 [ 7317.454199] LNet: Removed LNI 192.168.204.136@tcp31 [ 7317.455329] LNet: Removed LNI 192.168.204.136@tcp32 [ 7317.456451] LNet: Removed LNI 192.168.204.136@tcp33 [ 7317.459303] LNet: Removed LNI 192.168.204.136@tcp34 [ 7317.461499] LNet: Removed LNI 192.168.204.136@tcp35 [ 7317.462645] LNet: Removed LNI 192.168.204.136@tcp36 [ 7317.463835] LNet: Removed LNI 192.168.204.136@tcp37 [ 7317.464993] LNet: Removed LNI 192.168.204.136@tcp38 [ 7317.466155] LNet: Removed LNI 192.168.204.136@tcp39 [ 7317.467346] LNet: Removed LNI 192.168.204.136@tcp40 [ 7317.468474] LNet: Removed LNI 192.168.204.136@tcp41 [ 7317.469621] LNet: Removed LNI 192.168.204.136@tcp42 [ 7317.470762] LNet: Removed LNI 192.168.204.136@tcp43 [ 7317.471968] LNet: Removed LNI 192.168.204.136@tcp44 [ 7317.474305] LNet: Removed LNI 192.168.204.136@tcp45 [ 7317.476446] LNet: Removed LNI 192.168.204.136@tcp46 [ 7317.478353] LNet: Removed LNI 192.168.204.136@tcp47 [ 7317.480584] LNet: Removed LNI 192.168.204.136@tcp48 [ 7317.482442] LNet: Removed LNI 192.168.204.136@tcp49 [ 7317.490344] LNet: Removed LNI 192.168.204.136@tcp50 [ 7317.745453] Key type .llcrypt unregistered [ 7317.746518] Key type ._llcrypt unregistered [ 7323.882072] Key type ._llcrypt registered [ 7323.883064] Key type .llcrypt registered [ 7323.923626] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 7329.927433] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 7330.288688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7330.330856] alg: No test for adler32 (adler32-zlib) [ 7331.193041] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 7331.284232] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 7332.863152] Key type lgssc registered [ 7333.226808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7344.314517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7344.319949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7345.415771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7345.430655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7345.472813] Lustre: lustre-MDT0000: new disk, initializing [ 7345.498264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7345.506385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7346.851445] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7350.908726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7350.948368] Lustre: 504077:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7350.969277] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7350.971497] Lustre: Skipped 1 previous similar message [ 7351.006620] Lustre: lustre-MDT0001: new disk, initializing [ 7351.039311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7351.054814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7351.057970] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7352.450477] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7356.103241] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7357.866800] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7360.470472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7360.563931] Lustre: *** cfs_fail_loc=90f, val=0*** [ 7360.592150] Lustre: lustre-OST0000: new disk, initializing [ 7360.593711] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7360.595883] Lustre: 506092:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7360.620835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7362.605306] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7366.298521] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7366.644924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7366.648622] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7366.680500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7371.745735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7371.747054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7371.748307] LustreError: Skipped 1 previous similar message [ 7371.753746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7371.757075] Lustre: Skipped 1 previous similar message [ 7374.740124] Lustre: server umount lustre-OST0000 complete [ 7376.865039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7376.865495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7376.868431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7376.873221] Lustre: Skipped 2 previous similar messages [ 7376.876196] Lustre: Skipped 1 previous similar message [ 7381.984121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7381.988186] Lustre: Skipped 1 previous similar message [ 7382.172690] Lustre: server umount lustre-MDT0000 complete [ 7383.447186] LustreError: 504873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781668220 with bad export cookie 6162754927707217650 [ 7383.450528] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7383.454360] LustreError: 504873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7383.579633] Lustre: server umount lustre-MDT0001 complete [ 7387.326345] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 7388.446738] Key type lgssc unregistered [ 7388.571336] LNet: 507996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7388.573796] LNetError: 507996:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7388.586273] LNet: Removed LNI 192.168.204.136@tcp [ 7388.881504] Key type .llcrypt unregistered [ 7388.882668] Key type ._llcrypt unregistered [ 7396.224674] Key type ._llcrypt registered [ 7396.226010] Key type .llcrypt registered [ 7396.269464] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 7396.612614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7396.672029] alg: No test for adler32 (adler32-zlib) [ 7397.544778] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 7397.638470] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 7399.231180] Key type lgssc registered [ 7399.638179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7402.250685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7402.258021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7403.407551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7404.684061] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7407.480905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7407.618266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7408.963186] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7411.357195] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7413.170687] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7415.427553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7415.560295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7417.515815] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7419.942813] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7433.794294] loop0: detected capacity change from 0 to 204800000 [ 7433.858064] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7433.861357] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7433.864993] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7435.766172] loop0: detected capacity change from 0 to 204800000 [ 7435.824052] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7435.828714] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7435.832313] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7437.468216] loop0: detected capacity change from 0 to 204800000 [ 7439.299834] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7439.354026] Lustre: fs15246-OST0000: new disk, initializing [ 7439.356615] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7439.360467] Lustre: 512853:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7439.379267] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7441.297107] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7445.151916] loop1: detected capacity change from 0 to 204800000 [ 7447.046447] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7447.087756] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7447.108345] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7447.145615] Lustre: fs15246-MDT0000: new disk, initializing [ 7447.196886] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7447.208583] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7447.246420] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7448.287867] Lustre: 513975:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000e554d8f8 x1868214550166528/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781668301 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7448.544876] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7456.230289] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7457.375217] LustreError: 513975:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 7457.378604] LustreError: 513975:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 7457.381421] LustreError: 513975:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 7461.346856] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7461.842181] Lustre: server umount fs15246-MDT0000 complete [ 7464.864427] Lustre: server umount fs15246-OST0000 complete [ 7469.024579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7469.026778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7469.030902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7470.047891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7470.050336] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7471.585238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7471.587515] Lustre: Skipped 2 previous similar messages [ 7474.316222] Lustre: server umount lustre-OST0000 complete [ 7476.704936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7476.705735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7476.710076] Lustre: Skipped 1 previous similar message [ 7481.694918] Lustre: server umount lustre-MDT0000 complete [ 7481.824972] LustreError: 509981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7481.829547] LustreError: 509981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7483.013364] LustreError: 509224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781668320 with bad export cookie 16296441888640301 [ 7483.016717] LustreError: 509224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7483.017293] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7483.149253] Lustre: server umount lustre-MDT0001 complete [ 7486.925238] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 7488.045903] Key type lgssc unregistered [ 7488.194407] LNet: 517069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7488.196824] LNetError: 517069:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7488.205324] LNet: Removed LNI 192.168.204.136@tcp [ 7488.489737] Key type .llcrypt unregistered [ 7488.491145] Key type ._llcrypt unregistered [ 7495.734076] Key type ._llcrypt registered [ 7495.735246] Key type .llcrypt registered [ 7495.777381] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 7496.193834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7496.200439] alg: No test for adler32 (adler32-zlib) [ 7497.047403] Lustre: Lustre: Build Version: 2.17.54_40_g6e23802 [ 7497.125553] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 7498.703146] Key type lgssc registered [ 7499.091310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7511.135966] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 7514.040906] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 7525.336215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7525.341873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7526.431584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7526.442914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7526.479557] Lustre: lustre-MDT0000: new disk, initializing [ 7526.503746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7526.509830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7527.859153] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7532.021065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7532.062633] Lustre: 521503:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7532.084751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7532.086791] Lustre: Skipped 1 previous similar message [ 7532.126436] Lustre: lustre-MDT0001: new disk, initializing [ 7532.165216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7532.180643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7532.184686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7533.551579] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7537.358257] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7539.176983] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7541.413710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7541.520416] Lustre: lustre-OST0000: new disk, initializing [ 7541.522239] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7541.524969] Lustre: 523469:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7541.556456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7543.503066] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7543.736695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7543.739681] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7543.776173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7547.279050] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7548.753407] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7548.835956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7550.249636] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7550.323074] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7554.021136] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7554.022520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7554.023566] LustreError: Skipped 1 previous similar message [ 7554.028108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7554.030992] Lustre: Skipped 1 previous similar message [ 7559.138315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7559.140819] Lustre: Skipped 1 previous similar message [ 7564.257427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7564.259481] Lustre: Skipped 1 previous similar message [ 7565.279112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7565.338131] Lustre: server umount lustre-OST0000 complete [ 7568.351765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7568.353916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7568.358245] Lustre: Skipped 1 previous similar message [ 7568.360313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7569.377065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7572.761889] Lustre: server umount lustre-MDT0000 complete [ 7573.947407] LustreError: 521495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781668411 with bad export cookie 933143366959011269 [ 7573.950043] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7573.951334] LustreError: 521495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7574.080533] Lustre: server umount lustre-MDT0001 complete [ 7581.837500] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 7584.879593] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 7594.652211] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7479 sec ========== 23:53:51 (1781668431) [ 7595.196537] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:53:51 (1781668431) === [ 7596.257480] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:53:52 (1781668432) === [ 7609.143913] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 7610.069945] Key type lgssc unregistered [ 7610.195303] LNet: 530099:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7610.198934] LNetError: 530099:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7610.207314] LNet: Removed LNI 192.168.204.136@tcp [ 7610.479047] Key type .llcrypt unregistered [ 7610.480089] Key type ._llcrypt unregistered