[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 514435039 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003325] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.007663] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010142] LSM: Security Framework initializing [ 0.011067] Yama: becoming mindful. [ 0.012044] SELinux: Initializing. [ 0.013074] *** VALIDATE selinux *** [ 0.022323] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027514] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029084] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031116] *** VALIDATE tmpfs *** [ 0.032479] *** VALIDATE proc *** [ 0.033244] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035265] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037110] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039028] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.043281] debug: unmapping init [mem 0xffffffff9a859000-0xffffffff9a860fff] [ 0.046177] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047686] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048022] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050010] ... generic registers: 4 [ 0.050903] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052012] ... fixed-purpose events: 3 [ 0.052889] ... event mask: 000000070000000f [ 0.053295] rcu: Hierarchical SRCU implementation. [ 0.055195] smp: Bringing up secondary CPUs ... [ 0.056421] x86: Booting SMP configuration: [ 0.057027] .... node #0, CPUs: #1 #2 #3 [ 0.060403] smp: Brought up 1 node, 4 CPUs [ 0.062011] smpboot: Max logical packages: 1 [ 0.063014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145027] node 0 deferred pages initialised in 79ms [ 0.149088] devtmpfs: initialized [ 0.150192] x86/mm: Memory block size: 128MB [ 0.152793] gcov: version magic: 0x41383552 [ 0.154183] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157347] pinctrl core: initialized pinctrl subsystem [ 0.160261] [ 0.160943] ************************************************************* [ 0.163014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165011] ** ** [ 0.168013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170011] ** ** [ 0.172012] ** This means that this kernel is built to expose internal ** [ 0.175015] ** IOMMU data structures, which may compromise security on ** [ 0.177013] ** your system. ** [ 0.179013] ** ** [ 0.181012] ** If you see this message and you are not debugging the ** [ 0.184015] ** kernel, report this immediately to your vendor! ** [ 0.186017] ** ** [ 0.189013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.191010] ************************************************************* [ 0.193432] NET: Registered protocol family 16 [ 0.195546] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.199100] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.201088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.205150] cpuidle: using governor menu [ 0.206664] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.208052] PCI: Using configuration type 1 for base access [ 0.209091] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.219069] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.221123] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.224157] cryptd: max_cpu_qlen set to 1000 [ 0.227266] ACPI: Added _OSI(Module Device) [ 0.229019] ACPI: Added _OSI(Processor Device) [ 0.230017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.232020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.237908] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.243473] ACPI: Interpreter enabled [ 0.245077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.246017] ACPI: Using IOAPIC for interrupt routing [ 0.248140] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.250469] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.260633] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.264049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.266018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.269082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.273616] acpiphp: Slot [2] registered [ 0.275306] acpiphp: Slot [5] registered [ 0.277174] acpiphp: Slot [6] registered [ 0.278191] acpiphp: Slot [7] registered [ 0.279094] acpiphp: Slot [8] registered [ 0.279944] acpiphp: Slot [9] registered [ 0.281082] acpiphp: Slot [10] registered [ 0.281974] acpiphp: Slot [3] registered [ 0.283075] acpiphp: Slot [4] registered [ 0.284022] acpiphp: Slot [11] registered [ 0.285014] acpiphp: Slot [12] registered [ 0.285988] acpiphp: Slot [13] registered [ 0.287068] acpiphp: Slot [14] registered [ 0.288000] acpiphp: Slot [15] registered [ 0.288000] acpiphp: Slot [16] registered [ 0.289106] acpiphp: Slot [17] registered [ 0.290099] acpiphp: Slot [18] registered [ 0.290882] acpiphp: Slot [19] registered [ 0.292089] acpiphp: Slot [20] registered [ 0.293105] acpiphp: Slot [21] registered [ 0.294077] acpiphp: Slot [22] registered [ 0.294990] acpiphp: Slot [23] registered [ 0.296068] acpiphp: Slot [24] registered [ 0.296997] acpiphp: Slot [25] registered [ 0.298098] acpiphp: Slot [26] registered [ 0.299073] acpiphp: Slot [27] registered [ 0.300117] acpiphp: Slot [28] registered [ 0.301126] acpiphp: Slot [29] registered [ 0.303114] acpiphp: Slot [30] registered [ 0.304063] acpiphp: Slot [31] registered [ 0.304997] PCI host bridge to bus 0000:00 [ 0.306017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.307018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.309039] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.311038] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.314035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.317042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.319220] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.322564] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.324925] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.332014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.336039] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.337017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.340017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.343021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.345645] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.349733] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.353068] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.356831] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.362025] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.375021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.380021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.387169] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.399029] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.413032] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.447032] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.465698] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.480018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.487020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.513020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.522219] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.529028] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.541030] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.564028] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.576687] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.586016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.596017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.616021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.630044] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.641033] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.648024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.666030] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.679649] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.688018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.697029] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.719022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.732013] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.733242] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.734371] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.736211] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.737167] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.742313] iommu: Default domain type: Passthrough [ 0.744508] SCSI subsystem initialized [ 0.745104] ACPI: bus type USB registered [ 0.745951] usbcore: registered new interface driver usbfs [ 0.747057] usbcore: registered new interface driver hub [ 0.748082] usbcore: registered new device driver usb [ 0.749238] pps_core: LinuxPPS API ver. 1 registered [ 0.750012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.752064] PTP clock support registered [ 0.753185] EDAC MC: Ver: 3.0.0 [ 0.754332] PCI: Using ACPI for IRQ routing [ 0.757020] NetLabel: Initializing [ 0.757826] NetLabel: domain hash size = 128 [ 0.758015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.760097] NetLabel: unlabeled traffic allowed by default [ 0.761194] vgaarb: loaded [ 0.763336] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.764015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.769059] clocksource: Switched to clocksource kvm-clock [ 0.879919] VFS: Disk quotas dquot_6.6.0 [ 0.881716] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.884606] *** VALIDATE ramfs *** [ 0.885992] *** VALIDATE hugetlbfs *** [ 0.887793] pnp: PnP ACPI init [ 0.891120] pnp: PnP ACPI: found 6 devices [ 0.906789] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.909204] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.910704] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.912699] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.915228] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.918031] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.921071] NET: Registered protocol family 2 [ 0.923419] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.928535] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.932340] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.938459] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.942308] TCP: Hash tables configured (established 65536 bind 65536) [ 0.945681] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.948711] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.951364] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.955199] NET: Registered protocol family 1 [ 0.958307] RPC: Registered named UNIX socket transport module. [ 0.960433] RPC: Registered udp transport module. [ 0.961663] RPC: Registered tcp transport module. [ 0.962759] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.964633] NET: Registered protocol family 44 [ 0.966417] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.968392] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.970713] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.973271] PCI: CLS 0 bytes, default 64 [ 0.975327] Unpacking initramfs... [ 2.375162] debug: unmapping init [mem 0xffff946b3cc54000-0xffff946b3ffbffff] [ 2.379559] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.382177] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.385482] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.855498] Initialise system trusted keyrings [ 2.856837] Key type blacklist registered [ 2.858337] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.870207] zbud: loaded [ 2.873778] *** VALIDATE nfs *** [ 2.875183] *** VALIDATE nfs4 *** [ 2.876857] pstore: using deflate compression [ 2.880487] Platform Keyring initialized [ 2.978298] NET: Registered protocol family 38 [ 2.980224] Key type asymmetric registered [ 2.981802] Asymmetric key parser 'x509' registered [ 2.983586] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.986658] io scheduler mq-deadline registered [ 2.988264] io scheduler kyber registered [ 2.989871] io scheduler bfq registered [ 2.991592] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.994589] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.997240] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.000155] ACPI: Power Button [PWRF] [ 3.005876] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.013113] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.031581] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.047359] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.065746] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.091743] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.122613] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.127930] Non-volatile memory driver v1.3 [ 3.129888] Linux agpgart interface v0.103 [ 3.160358] virtio_blk virtio1: [vda] 146152 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.163264] vda: detected capacity change from 0 to 74829824 [ 3.183079] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.185314] vdb: detected capacity change from 0 to 1073741824 [ 3.205365] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.208561] vdc: detected capacity change from 0 to 2621440000 [ 3.224633] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.226930] vdd: detected capacity change from 0 to 2621440000 [ 3.245384] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.248367] vde: detected capacity change from 0 to 4294967296 [ 3.263440] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.265195] vdf: detected capacity change from 0 to 4294967296 [ 3.276250] libphy: Fixed MDIO Bus: probed [ 3.282470] usbcore: registered new interface driver usbserial_generic [ 3.284899] usbserial: USB Serial support registered for generic [ 3.286954] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.291484] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.293073] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.295151] mousedev: PS/2 mouse device common for all mice [ 3.298037] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.299439] rtc_cmos 00:05: RTC can wake from S4 [ 3.304354] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.307255] rtc_cmos 00:05: registered as rtc0 [ 3.310344] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.310752] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.316848] intel_pstate: CPU model not supported [ 3.320078] hid: raw HID events driver (C) Jiri Kosina [ 3.322029] usbcore: registered new interface driver usbhid [ 3.324010] usbhid: USB HID core driver [ 3.325634] drop_monitor: Initializing network drop monitor service [ 3.328146] Initializing XFRM netlink socket [ 3.330299] NET: Registered protocol family 10 [ 3.332804] Segment Routing with IPv6 [ 3.334364] NET: Registered protocol family 17 [ 3.336543] mpls_gso: MPLS GSO support [ 3.344114] RAS: Correctable Errors collector initialized. [ 3.346460] AVX version of gcm_enc/dec engaged. [ 3.348257] AES CTR mode by8 optimization enabled [ 3.423694] sched_clock: Marking stable (3423670614, 0)->(4341404834, -917734220) [ 3.425894] registered taskstats version 1 [ 3.427359] Loading compiled-in X.509 certificates [ 3.428931] zswap: loaded using pool lzo/zbud [ 3.450194] Key type big_key registered [ 3.462643] Key type encrypted registered [ 3.463869] ima: No TPM chip found, activating TPM-bypass! [ 3.465628] ima: Allocated hash algorithm: sha1 [ 3.467189] ima: No architecture policies found [ 3.468802] evm: Initialising EVM extended attributes: [ 3.470638] evm: security.selinux [ 3.471929] evm: security.ima [ 3.472893] evm: security.capability [ 3.474058] evm: HMAC attrs: 0x1 [ 3.476145] rtc_cmos 00:05: setting system clock to 2026-08-24 12:19:38 UTC (1787573978) [ 3.481705] debug: unmapping init [mem 0xffffffff9b803000-0xffffffff9b9fffff] [ 3.483924] debug: unmapping init [mem 0xffffffff9a582000-0xffffffff9a858fff] [ 3.491153] Write protecting the kernel read-only data: 28672k [ 3.493494] debug: unmapping init [mem 0xffffffff98c03000-0xffffffff98dfffff] [ 3.495656] debug: unmapping init [mem 0xffffffff99514000-0xffffffff995fffff] [ 3.523211] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.527866] systemd[1]: Detected virtualization kvm. [ 3.528973] systemd[1]: Detected architecture x86-64. [ 3.529989] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.551187] systemd[1]: No hostname configured. [ 3.552860] systemd[1]: Set hostname to . [ 3.554865] random: systemd: uninitialized urandom read (16 bytes read) [ 3.557211] systemd[1]: Initializing machine ID from random generator. [ 3.607370] random: ln: uninitialized urandom read (6 bytes read) [ 3.695133] random: systemd: uninitialized urandom read (16 bytes read) [ 3.697307] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.704702] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 3.707657] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.316448] device-mapper: uevent: version 1.0.3 [ 4.317937] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ 4.952047] random: fast init done [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.047969] virtio_net virtio0 ens2: renamed from eth0 [ 5.068274] scsi host0: ata_piix [ 5.074262] scsi host1: ata_piix [ 5.075957] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.078302] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.465540] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.979126] random: crng init done [ 9.980713] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.418854] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.621253] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.856980] SELinux: Disabled at runtime. [ 11.921312] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.929939] systemd[1]: Detected virtualization kvm. [ 11.931634] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.424810] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.427993] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.436035] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.440700] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.444659] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.452369] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.457221] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File Syste[ 12.489594] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.873826] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.137114] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.138361] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.292102] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.311196] EDAC sbridge: Ver: 1.1.2 [ 14.955419] Key type dns_resolver registered [ 15.246302] NFS: Registering the id_resolver key type [ 15.248590] Key type id_resolver registered [ 15.250247] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg216-server login: [ 40.594586] libcfs: loading out-of-tree module taints kernel. [ 40.616108] Key type ._llcrypt registered [ 40.618058] Key type .llcrypt registered [ 40.675605] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 48.831730] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 49.616336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 49.627368] alg: No test for adler32 (adler32-zlib) [ 50.679401] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 51.032352] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 52.679207] Key type lgssc registered [ 53.430743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 62.975817] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 111.144826] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 125.438578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 125.481381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 126.968624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 127.028244] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 127.134470] Lustre: lustre-MDT0000: new disk, initializing [ 127.263094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 127.282321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 132.377153] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 140.593170] hrtimer: interrupt took 2211091 ns [ 147.760060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.925192] Lustre: 6483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 147.957754] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 147.968941] Lustre: Skipped 1 previous similar message [ 148.045981] Lustre: lustre-MDT0001: new disk, initializing [ 148.111990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 148.152385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 148.167615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 153.294700] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 159.321658] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 170.699162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 171.180393] Lustre: lustre-OST0000: new disk, initializing [ 171.187324] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 171.197411] Lustre: 8423:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 171.309462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 178.382698] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.775655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 178.784629] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 178.903378] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 194.094676] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 194.271645] Lustre: lustre-OST0001: new disk, initializing [ 194.275603] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 194.283758] Lustre: 9494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 194.399458] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 201.282707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 201.319239] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 201.496123] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 202.362581] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.125570] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 226.497540] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 234.329643] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing check_logdir /tmp/testlogs/ [ 240.431295] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing yml_node [ 245.512990] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 248.074809] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 250.588810] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 252.911629] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 08:23:46 EDT 2026 [ 274.557732] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 276.212099] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 293.857840] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.862406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 293.875089] Lustre: Skipped 1 previous similar message [ 293.890963] Lustre: Skipped 3 previous similar messages [ 298.976996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 298.987790] Lustre: Skipped 3 previous similar messages [ 299.416187] Lustre: server umount lustre-MDT0000 complete [ 303.437741] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787574278 with bad export cookie 10451068286976671852 [ 303.463978] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 303.466244] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 304.097455] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.107257] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 304.112392] Lustre: Skipped 3 previous similar messages [ 304.139914] Lustre: Skipped 1 previous similar message [ 308.130650] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 310.062423] Lustre: server umount lustre-MDT0001 complete [ 315.555332] Lustre: server umount lustre-OST0000 complete [ 321.131441] Lustre: server umount lustre-OST0001 complete [ 330.068235] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 342.608319] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 391.735262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 392.217171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 392.245270] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 392.309510] Lustre: lustre-MDT0000: new disk, initializing [ 392.380759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 392.395256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 396.670831] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 410.357400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.585334] Lustre: 15052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 410.730925] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 410.736716] Lustre: Skipped 1 previous similar message [ 410.849402] Lustre: lustre-MDT0001: new disk, initializing [ 411.047556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 411.102600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 411.115128] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 416.616407] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 427.016671] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 433.858758] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 441.776599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 442.083991] Lustre: lustre-OST0000: new disk, initializing [ 442.087963] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 442.095814] Lustre: 17024:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 442.212726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 443.660338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 443.676984] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 443.945415] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 448.759407] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 460.305371] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 467.495966] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 467.786118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 472.963572] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 473.169717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 479.717367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 479.724875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 479.753109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 479.786019] Lustre: Skipped 2 previous similar messages [ 481.658051] Lustre: server umount lustre-OST0000 complete [ 487.905065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 487.913746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 487.927989] Lustre: Skipped 1 previous similar message [ 487.935266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 500.191673] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 500.744570] Lustre: server umount lustre-MDT0000 complete [ 504.290085] LustreError: 15063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.320957] LustreError: 15063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 505.407951] LustreError: 15043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787574480 with bad export cookie 10451068286976673385 [ 505.415966] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 505.418393] LustreError: 15043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 505.743799] Lustre: server umount lustre-MDT0001 complete [ 513.519389] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 08:28:06 (1787574486) [ 534.368539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.694028] LustreError: 19901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.773143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 539.373688] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 540.136825] LustreError: 19902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.226628] LustreError: 19901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.347867] LustreError: 19902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.791223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 554.781474] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 564.834783] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 570.750666] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 573.453255] Lustre: Failing over lustre-MDT0000 [ 573.893605] Lustre: server umount lustre-MDT0000 complete [ 575.593904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 575.625995] Lustre: Skipped 1 previous similar message [ 575.640719] LustreError: 19901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 575.662727] LustreError: 19901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 591.263151] Lustre: 3619:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787574550/real 1787574550] req@ffff946bc1b9ce00 x1874407222054656/t0(0) o400->MGC192.168.202.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787574566 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 591.300641] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 594.501457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 595.425665] LustreError: 19896:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.443414] LustreError: 19896:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 601.570898] LustreError: 3617:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff946a83378e00 x1874407222057472/t0(0) o250->MGC192.168.202.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 601.890751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 601.898611] Lustre: Skipped 1 previous similar message [ 601.951751] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 606.914921] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 607.224537] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 607.237875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 607.277323] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 618.458535] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 625.636787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 625.652342] Lustre: Skipped 1 previous similar message [ 625.659451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 625.667847] Lustre: Skipped 7 previous similar messages [ 637.409404] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 637.799724] Lustre: server umount lustre-MDT0000 complete [ 640.999588] LustreError: 19896:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 641.023553] LustreError: 19896:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 642.558732] LustreError: 19881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787574617 with bad export cookie 10451068286976674799 [ 642.563398] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 642.569190] LustreError: 19881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 642.847678] Lustre: server umount lustre-MDT0001 complete [ 653.750290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 659.853142] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 669.489142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 674.800492] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 684.073951] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 690.789813] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 693.425896] Lustre: Failing over lustre-MDT0000 [ 693.704818] Lustre: server umount lustre-MDT0000 complete [ 694.510318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 694.524024] Lustre: Skipped 1 previous similar message [ 710.122630] LustreError: 24246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 710.158785] LustreError: 24246:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 710.627332] Lustre: 3621:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787574669/real 1787574669] req@ffff946b8de3bb80 x1874407222085504/t0(0) o400->MGC192.168.202.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787574685 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 710.648496] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 714.393506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 721.897159] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x9109a73c20984f99 [ 721.915226] Lustre: MGC192.168.202.116@tcp: Connection restored to 0@lo (at 0@lo) [ 721.929261] Lustre: Skipped 1 previous similar message [ 722.255685] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 726.868481] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 727.523518] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 727.536267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 727.577768] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 736.751429] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 742.881612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 742.891900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 742.912729] Lustre: Skipped 1 previous similar message [ 742.921979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 742.938753] Lustre: Skipped 4 previous similar messages [ 747.530567] Lustre: server umount lustre-MDT0000 complete [ 751.804711] LustreError: 23223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787574726 with bad export cookie 10451068286976675737 [ 751.812509] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 751.835780] LustreError: 23223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 752.126245] Lustre: server umount lustre-MDT0001 complete [ 763.033375] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 08:32:16 (1787574736) [ 782.372771] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 792.252576] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 831.731449] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 843.902393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 844.234441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 844.296642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 844.404733] Lustre: lustre-MDT0000: new disk, initializing [ 844.494202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 844.503719] Lustre: Skipped 3 previous similar messages [ 844.545035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 849.303637] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 861.781830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 861.895288] Lustre: 29621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 862.408602] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 867.697271] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 877.615408] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 884.396545] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 891.237826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 891.425230] Lustre: 31588:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 893.312561] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 893.356251] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 893.434941] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 893.466635] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 897.997604] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 908.612658] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 917.105564] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 934.372078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 934.384839] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 934.389516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 934.398501] Lustre: Skipped 2 previous similar messages [ 934.411625] Lustre: Skipped 3 previous similar messages [ 939.669542] Lustre: server umount lustre-OST0000 complete [ 944.099927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 944.111959] LustreError: Skipped 1 previous similar message [ 950.251516] Lustre: server umount lustre-MDT0000 complete [ 953.975515] LustreError: 29614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787574928 with bad export cookie 10451068286976676150 [ 953.983915] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 953.987799] LustreError: 29614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 954.022676] LustreError: 3620:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 954.423209] Lustre: server umount lustre-MDT0001 complete [ 967.505529] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 970.625395] Key type lgssc unregistered [ 971.088600] LNet: 33574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.092485] LNetError: 33574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.122929] LNet: Removed LNI 192.168.202.116@tcp [ 972.413207] Key type .llcrypt unregistered [ 972.414724] Key type ._llcrypt unregistered [ 998.028920] Key type ._llcrypt registered [ 998.033543] Key type .llcrypt registered [ 998.210176] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 999.534248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 999.773096] alg: No test for adler32 (adler32-zlib) [ 1001.017526] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 1001.355039] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1003.087296] Key type lgssc registered [ 1004.393735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1014.299362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1014.342328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1016.254353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1022.351995] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1026.533667] LustreError: 34989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1026.561656] LustreError: 34989:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1031.394980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1031.648248] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1031.783080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1037.062835] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1044.199840] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1050.085075] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1057.320672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1057.915194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1063.592520] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1070.302152] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1076.353197] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1088.492544] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 08:37:41 (1787575061) [ 1090.544152] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1093.049918] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1101.334874] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 08:37:54 (1787575074) [ 1113.452321] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 08:38:06 (1787575086) [ 1114.590980] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1124.204564] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1124.219074] Lustre: Skipped 2 previous similar messages [ 1125.572330] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 08:38:19 (1787575099) [ 1129.424958] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1129.429458] Lustre: Skipped 3 previous similar messages [ 1147.106611] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1147.111603] Lustre: Skipped 6 previous similar messages [ 1149.994642] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 08:38:43 (1787575123) [ 1152.853525] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1156.779857] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1166.154556] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 08:38:59 (1787575139) [ 1168.231686] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1168.238297] Lustre: Skipped 3 previous similar messages [ 1181.675662] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 08:39:15 (1787575155) [ 1195.491573] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1195.500551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1195.519275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1199.585902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1199.601952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1199.606388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1199.637539] Lustre: Skipped 1 previous similar message [ 1204.716439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1204.724146] Lustre: Skipped 1 previous similar message [ 1209.312226] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1209.498053] Lustre: server umount lustre-OST0000 complete [ 1214.945808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1214.947472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1214.959751] Lustre: Skipped 1 previous similar message [ 1214.971235] Lustre: Skipped 1 previous similar message [ 1219.415417] Lustre: server umount lustre-MDT0000 complete [ 1220.073946] LustreError: 34990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1220.101295] LustreError: 34990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1222.899571] LustreError: 34970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575197 with bad export cookie 5364091495748282206 [ 1222.902210] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1222.912679] LustreError: 34970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1223.128334] Lustre: server umount lustre-MDT0001 complete [ 1234.107846] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1236.335622] Key type lgssc unregistered [ 1236.591054] LNet: 42851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1236.595532] LNetError: 42851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1236.610854] LNet: Removed LNI 192.168.202.116@tcp [ 1237.182606] Key type .llcrypt unregistered [ 1237.185907] Key type ._llcrypt unregistered [ 1255.050606] Key type ._llcrypt registered [ 1255.064934] Key type .llcrypt registered [ 1255.214943] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 1272.467770] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1273.570725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1273.591330] alg: No test for adler32 (adler32-zlib) [ 1274.670660] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 1275.006184] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1276.712641] Key type lgssc registered [ 1278.002981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1318.989962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1319.046292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1320.339552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1320.402926] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1320.575606] Lustre: lustre-MDT0000: new disk, initializing [ 1320.727507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1320.754899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1326.284618] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1336.682255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1336.805398] Lustre: 46008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1336.851743] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1336.856122] Lustre: Skipped 1 previous similar message [ 1336.913439] Lustre: lustre-MDT0001: new disk, initializing [ 1337.019251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1337.064365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1337.094312] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1342.359862] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1351.996486] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1358.966577] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1366.397514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1366.719539] Lustre: lustre-OST0000: new disk, initializing [ 1366.722158] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1366.738453] Lustre: 47977:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1366.853036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1373.592590] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1375.787607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1375.794642] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1375.865917] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1383.054082] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1387.650167] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1387.878862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1392.944376] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1393.190282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1396.193392] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1396.201201] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1396.209170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1396.220120] Lustre: Skipped 1 previous similar message [ 1401.567378] Lustre: server umount lustre-OST0000 complete [ 1408.992744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1409.004348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1409.025550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1409.036458] Lustre: Skipped 1 previous similar message [ 1411.558569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1411.564523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1416.674358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1416.677165] Lustre: Skipped 1 previous similar message [ 1421.280996] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1421.470410] Lustre: server umount lustre-MDT0000 complete [ 1421.793763] LustreError: 47029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1421.820655] LustreError: 47029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1426.004946] LustreError: 47976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575401 with bad export cookie 1167584509064895179 [ 1426.012112] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1426.026437] LustreError: 47976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1426.879655] Lustre: server umount lustre-MDT0001 complete [ 1438.030288] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 08:43:31 (1787575411) [ 1449.145703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1449.754152] LustreError: 50440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1449.858578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1455.079770] LustreError: 50441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1455.702150] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1459.185860] LustreError: 50440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1464.300471] LustreError: 50441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1465.494516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1466.004464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1471.102628] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1479.551617] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1485.702555] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1492.204487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1492.555105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1498.340079] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1505.496686] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1513.230495] LustreError: 52968:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1514.177850] Lustre: Modifying parameter general.timeout=1 in log params [ 1515.989291] Lustre: Modifying parameter general.timeout=3 in log params [ 1515.993648] Lustre: Skipped 1 previous similar message [ 1518.693496] Lustre: Modifying parameter general.timeout=6 in log params [ 1518.699165] Lustre: Skipped 2 previous similar messages [ 1523.343336] Lustre: Modifying parameter general.timeout=11 in log params [ 1523.349609] Lustre: Skipped 4 previous similar messages [ 1531.952341] Lustre: Modifying parameter general.timeout=21 in log params [ 1531.956120] Lustre: Skipped 9 previous similar messages [ 1548.087088] Lustre: Modifying parameter general.timeout=38 in log params [ 1548.096585] Lustre: Skipped 16 previous similar messages [ 1581.022825] Lustre: Modifying parameter general.timeout=71 in log params [ 1581.026731] Lustre: Skipped 32 previous similar messages [ 1648.401961] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 08:47:01 (1787575621) [ 1649.533463] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1649.538062] Lustre: Skipped 59 previous similar messages [ 1652.187676] Lustre: 59917:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1654.209990] Lustre: 60016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1657.643564] Lustre: 60228:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1657.659443] Lustre: 60228:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1694.560390] Lustre: 61789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1694.566869] Lustre: 61789:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1694.594078] LustreError: 61789:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1698.882758] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 08:47:52 (1787575672) [ 1711.591458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1711.599956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1711.610910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1717.205712] Lustre: server umount lustre-MDT0000 complete [ 1717.736442] LustreError: 50441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1717.758071] LustreError: 50441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1721.286738] LustreError: 53017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575696 with bad export cookie 1167584509064896061 [ 1721.287697] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1721.302264] LustreError: 53017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1721.637312] Lustre: server umount lustre-MDT0001 complete [ 1735.688678] Lustre: server umount lustre-OST0000 complete [ 1760.382041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1760.805145] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1761.029478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1765.659170] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1773.944357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1774.033200] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1774.057481] Lustre: 64320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1779.406233] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1788.720125] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1797.331888] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1806.655749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1806.943094] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1807.196811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1807.200174] Lustre: Skipped 1 previous similar message [ 1813.604214] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1822.509051] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1824.422339] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1824.428813] Lustre: Skipped 22 previous similar messages [ 1824.458052] Lustre: 66886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1831.913349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1831.925807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1831.938993] Lustre: Skipped 2 previous similar messages [ 1831.956047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1831.961326] Lustre: Skipped 3 previous similar messages [ 1836.515510] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1837.747860] Lustre: server umount lustre-OST0000 complete [ 1841.647052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1841.665293] Lustre: Skipped 1 previous similar message [ 1846.241326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1847.788948] Lustre: server umount lustre-MDT0000 complete [ 1851.584792] LustreError: 64312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575826 with bad export cookie 1167584509064899162 [ 1851.589160] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1851.598364] LustreError: 64312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1852.022361] Lustre: server umount lustre-MDT0001 complete [ 1863.613768] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1866.186357] Key type lgssc unregistered [ 1866.519008] LNet: 68096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1866.525876] LNetError: 68096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1866.544534] LNet: Removed LNI 192.168.202.116@tcp [ 1867.238440] Key type .llcrypt unregistered [ 1867.240897] Key type ._llcrypt unregistered [ 1893.824349] Key type ._llcrypt registered [ 1893.831070] Key type .llcrypt registered [ 1893.920673] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1895.049177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1895.066446] alg: No test for adler32 (adler32-zlib) [ 1896.158457] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 1896.445057] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1898.175159] Key type lgssc registered [ 1899.645108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1908.496978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1908.525047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1910.213713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1915.385685] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1920.488886] LustreError: 69509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1920.507028] LustreError: 69509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1925.361620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1925.763996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1931.226640] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1939.802505] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1946.378708] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1955.617649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1956.151125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1963.365261] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1973.765291] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1982.874524] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1996.258245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1996.285737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1996.297703] Lustre: Skipped 2 previous similar messages [ 1998.647088] Lustre: server umount lustre-MDT0000 complete [ 2001.382695] LustreError: 70806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2001.394247] LustreError: 70806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2001.753719] LustreError: 69490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575976 with bad export cookie 1272599305825499559 [ 2001.761873] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2001.767296] LustreError: 69490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2002.063354] Lustre: server umount lustre-MDT0001 complete [ 2016.410859] Lustre: server umount lustre-OST0000 complete [ 2046.489711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2046.714970] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2046.745531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2047.170528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2052.033580] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2061.050502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2061.131575] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2061.148224] Lustre: 74384:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2061.574327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2066.156737] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2074.422383] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2081.136916] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2088.056414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2088.220186] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2088.644761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2094.439704] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2101.335120] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2103.229831] Lustre: 76962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2105.339213] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2110.947187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2110.952259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2110.963375] Lustre: Skipped 2 previous similar messages [ 2110.972398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2111.466059] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2116.486095] Lustre: server umount lustre-OST0000 complete [ 2121.699435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2121.716107] Lustre: Skipped 1 previous similar message [ 2121.719719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2121.722927] Lustre: Skipped 2 previous similar messages [ 2126.650273] Lustre: server umount lustre-MDT0000 complete [ 2126.824085] LustreError: 75305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2126.849502] LustreError: 75305:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2130.148357] LustreError: 74378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576105 with bad export cookie 1272599305825500406 [ 2130.151739] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2130.159016] LustreError: 74378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2130.491793] Lustre: server umount lustre-MDT0001 complete [ 2143.212719] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2146.259634] Key type lgssc unregistered [ 2146.560735] LNet: 78180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2146.574190] LNetError: 78180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2146.610716] LNet: Removed LNI 192.168.202.116@tcp [ 2147.467640] Key type .llcrypt unregistered [ 2147.469916] Key type ._llcrypt unregistered [ 2174.657833] Key type ._llcrypt registered [ 2174.659444] Key type .llcrypt registered [ 2174.740560] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2175.963323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2175.988489] alg: No test for adler32 (adler32-zlib) [ 2177.191238] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 2177.528610] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2179.248079] Key type lgssc registered [ 2180.463167] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2188.090585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2188.142786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2189.795722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2194.913827] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2199.010826] LustreError: 79594:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2199.034674] LustreError: 79594:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2204.128536] LustreError: 79595:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.465411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2205.370358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2209.803704] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2217.286113] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2224.104479] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2231.879301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2232.336313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2238.724171] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2246.089617] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2255.392791] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2256.326974] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2257.785632] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2257.790185] Lustre: Skipped 1 previous similar message [ 2259.981981] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2259.986782] Lustre: Skipped 2 previous similar messages [ 2264.193415] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2264.199143] Lustre: Skipped 4 previous similar messages [ 2272.556856] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2272.560834] Lustre: Skipped 8 previous similar messages [ 2289.060378] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2289.064529] Lustre: Skipped 16 previous similar messages [ 2321.692213] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2321.703112] Lustre: Skipped 36 previous similar messages [ 2357.302838] Lustre: 87692:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2360.033304] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 08:58:53 (1787576333) [ 2361.611870] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2363.511613] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 08:58:57 (1787576337) [ 2372.580191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2372.587711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2372.592434] LustreError: Skipped 1 previous similar message [ 2372.598097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2372.658330] Lustre: Skipped 1 previous similar message [ 2373.850953] Lustre: server umount lustre-OST0000 complete [ 2379.236819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2379.247183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2379.282032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2379.289357] Lustre: Skipped 1 previous similar message [ 2382.817361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2382.819688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2385.276474] Lustre: server umount lustre-MDT0000 complete [ 2387.938511] LustreError: 79594:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2387.957779] LustreError: 79594:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2389.958953] LustreError: 79574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576364 with bad export cookie 13491217692477337656 [ 2389.970631] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2390.981798] Lustre: server umount lustre-MDT0001 complete [ 2405.691645] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2409.458833] Key type lgssc unregistered [ 2409.932273] LNet: 88946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2409.938325] LNetError: 88946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2409.974727] LNet: Removed LNI 192.168.202.116@tcp [ 2411.268801] Key type .llcrypt unregistered [ 2411.271806] Key type ._llcrypt unregistered [ 2417.442619] Key type ._llcrypt registered [ 2417.444528] Key type .llcrypt registered [ 2417.538831] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2421.940740] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules [ 2422.791410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2422.804510] alg: No test for adler32 (adler32-zlib) [ 2423.953406] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 2424.277617] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2425.911914] Key type lgssc registered [ 2427.164600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2428.297362] LustreError: 90149:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2446.834642] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2450.808830] LustreError: 90149:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2455.651142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2455.669725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2457.067519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2461.214339] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2467.303732] LustreError: 91173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2467.328122] LustreError: 91173:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2470.004482] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:00:43 (1787576443) [ 2472.417640] LustreError: 91174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2474.946384] LustreError: 91205:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2474.958836] LustreError: 91205:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2474.967341] LustreError: 91205:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2475.401031] Lustre: server umount lustre-MDT0000 complete [ 2488.495489] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2490.982257] Key type lgssc unregistered [ 2491.249978] LNet: 92714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2491.258533] LNetError: 92714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2491.276865] LNet: Removed LNI 192.168.202.116@tcp [ 2492.083213] Key type .llcrypt unregistered [ 2492.088234] Key type ._llcrypt unregistered [ 2509.565429] Key type ._llcrypt registered [ 2509.567631] Key type .llcrypt registered [ 2509.724732] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2511.290986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2511.354851] alg: No test for adler32 (adler32-zlib) [ 2512.424414] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 2512.691979] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2514.375460] Key type lgssc registered [ 2515.369122] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2522.361165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2522.416633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2524.028219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2528.927143] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2533.352320] LustreError: 93934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2533.398717] LustreError: 93934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2537.915428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2538.490925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2543.827925] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2552.397485] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2558.974752] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2566.933948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2567.736667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2574.471894] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2582.736934] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2632.908248] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:03:26 (1787576606) [ 2634.930394] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2637.357293] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:03:30 (1787576610) [ 2644.449957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2644.472878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2644.479924] Lustre: Skipped 1 previous similar message [ 2644.502296] Lustre: Skipped 2 previous similar messages [ 2648.388279] Lustre: server umount lustre-MDT0000 complete [ 2649.574305] LustreError: 94901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2649.608382] LustreError: 94901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2652.108840] LustreError: 93913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576627 with bad export cookie 7672346878301130752 [ 2652.116554] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2652.465727] Lustre: server umount lustre-MDT0001 complete [ 2666.378368] Lustre: server umount lustre-OST0000 complete [ 2677.268546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2677.778394] LustreError: 98208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.063386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2682.386841] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2683.397585] LustreError: 98209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2683.424428] LustreError: 98209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2687.521573] LustreError: 98208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2690.968724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2691.570685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2697.068266] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2705.444399] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2711.552816] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2727.947815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2728.160087] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2728.166065] LustreError: 98196:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2728.189935] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2728.196724] LustreError: 100292:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2728.205985] LustreError: 100292:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2728.212803] LustreError: 100292:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2728.268820] Lustre: server umount lustre-OST0000 complete [ 2728.275115] LustreError: 100292:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2737.371701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2737.633972] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2737.653708] LustreError: 98197:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2737.684935] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2737.708111] LustreError: 100759:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2737.726958] LustreError: 100759:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2737.739690] LustreError: 100759:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2737.832652] Lustre: server umount lustre-OST0000 complete [ 2737.838857] LustreError: 100759:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2750.083841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2750.205418] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2750.209257] Lustre: Found index 0 for lustre-OST0000, updating log [ 2750.214573] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2750.290238] Lustre: lustre-OST0000: new disk, initializing [ 2750.296364] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2750.304061] Lustre: 101296:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2750.639434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2756.627349] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2760.587974] Lustre: Failing over lustre-OST0000 [ 2760.776278] Lustre: server umount lustre-OST0000 complete [ 2766.817990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2766.836415] Lustre: Skipped 1 previous similar message [ 2766.842622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2766.874034] Lustre: Skipped 1 previous similar message [ 2771.819539] Lustre: server umount lustre-MDT0000 complete [ 2771.936571] LustreError: 98205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2771.955690] LustreError: 98205:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2777.177136] LustreError: 98188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576752 with bad export cookie 7672346878301133496 [ 2777.212131] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2778.257877] Lustre: server umount lustre-MDT0001 complete [ 2788.251181] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:06:01 (1787576761) [ 2804.532450] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2818.512626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2819.311752] LustreError: 103641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2819.344686] LustreError: 103641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2819.545097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2825.745794] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2834.960175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2840.941295] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2844.252852] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2851.045570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2851.898584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2851.904199] Lustre: Skipped 1 previous similar message [ 2857.674881] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2866.260856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2866.383572] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2866.452240] Lustre: lustre-OST0001: new disk, initializing [ 2866.454898] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2866.462158] Lustre: 105960:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2872.926710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2872.932048] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2872.983758] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2873.043869] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2881.886120] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2889.493181] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2896.224955] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.16@tcp (stopping) [ 2897.376272] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2897.384398] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2898.914210] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2898.918305] Lustre: Skipped 1 previous similar message [ 2900.433548] Lustre: server umount lustre-MDT0001 complete [ 2901.300909] LustreError: 106561:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2901.337754] LustreError: 106561:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2911.319376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2911.478648] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2911.484647] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2911.537414] Lustre: 103630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2913.425061] LustreError: 107523:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2913.434290] LustreError: 107523:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2913.443195] LustreError: 103634:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2913.571425] LustreError: 107538:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.116@tcp: cfg command failed: rc = -17 [ 2913.586684] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2913.586684] [ 2917.155816] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2917.387930] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2929.702481] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 09:08:23 (1787576903) [ 2948.923327] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2967.819572] Lustre: 109578:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2990.295881] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2994.994980] Lustre: 110713:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3085.793930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3085.809739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3085.828064] Lustre: Skipped 2 previous similar messages [ 3085.833982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3085.848145] Lustre: Skipped 2 previous similar messages [ 3087.829565] Lustre: server umount lustre-MDT0000 complete [ 3091.354263] LustreError: 103623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577066 with bad export cookie 7672346878301134490 [ 3091.356103] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3091.361193] LustreError: 103623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3091.425608] LustreError: 105159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3091.430374] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3091.436221] LustreError: 105159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 3091.453507] Lustre: Skipped 4 previous similar messages [ 3091.637278] Lustre: server umount lustre-MDT0001 complete [ 3098.048302] Lustre: server umount lustre-OST0000 complete [ 3102.215878] Lustre: server umount lustre-OST0001 complete [ 3109.435053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3120.111801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3129.439830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3142.190132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3165.664626] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3175.760262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3175.779197] Lustre: lustre-MDT0000: reset Object Index mappings [ 3176.446028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3176.449082] Lustre: Skipped 2 previous similar messages [ 3182.208541] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3193.574994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3193.598911] Lustre: lustre-MDT0001: reset Object Index mappings [ 3201.703325] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3205.157647] Lustre: 115816:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3214.477221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3219.270990] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3221.294559] LustreError: 116169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3221.332444] LustreError: 116169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3221.349427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3222.713294] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3235.522260] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3242.010837] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3242.019754] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3245.571605] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3254.874412] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3259.239298] Lustre: 117693:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3319.450032] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:14:52 (1787577292) [ 3344.354618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3344.360838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3344.373940] Lustre: Skipped 2 previous similar messages [ 3344.393776] Lustre: Skipped 9 previous similar messages [ 3349.520823] Lustre: server umount lustre-MDT0000 complete [ 3353.198455] LustreError: 114648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577328 with bad export cookie 7672346878301181299 [ 3353.202901] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3353.213556] LustreError: 114648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3359.749868] Lustre: server umount lustre-MDT0001 complete [ 3378.143135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3378.323417] Lustre: server umount lustre-OST0000 complete [ 3388.025412] Lustre: server umount lustre-OST0001 complete [ 3395.909452] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 3412.620422] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3452.627736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3452.924997] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3452.956202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3453.098778] Lustre: lustre-MDT0000: new disk, initializing [ 3453.233683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3453.239985] Lustre: Skipped 3 previous similar messages [ 3453.254763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3458.714924] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3465.546869] Lustre: server umount lustre-MDT0000 complete [ 3475.202287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3475.744194] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3475.761227] Lustre: 121999:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3475.780619] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3480.732857] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3486.434724] Lustre: server umount lustre-MDT0000 complete [ 3498.444920] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:17:51 (1787577471) [ 3501.717849] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3503.545675] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 09:17:57 (1787577477) [ 3526.725533] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3538.884057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3545.790730] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3556.655833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3556.847959] Lustre: 124051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3556.948661] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3556.951489] Lustre: Skipped 1 previous similar message [ 3557.075434] Lustre: lustre-MDT0001: new disk, initializing [ 3557.332776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3557.364963] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3562.971382] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3568.976402] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3577.818782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3578.120237] Lustre: lustre-OST0000: new disk, initializing [ 3578.125730] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3578.156680] Lustre: 125629:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3579.614318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3579.626093] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3579.718268] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3586.121206] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3600.300181] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3600.783230] Lustre: lustre-OST0001: new disk, initializing [ 3600.795179] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3600.812518] Lustre: 126502:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3601.988437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3601.997633] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3602.156441] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3608.978784] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3620.273044] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3625.294419] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4056.833533] Lustre: 132926:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4056.856198] Lustre: 132926:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4056.877607] Lustre: 132926:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4056.888081] Lustre: 132926:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4056.901919] Lustre: 132926:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4056.909238] Lustre: 132926:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4360.480793] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:32:13 (1787578333) [ 4378.597949] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4378.610774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4378.613813] Lustre: Skipped 5 previous similar messages [ 4378.636745] Lustre: Skipped 7 previous similar messages [ 4383.723892] LustreError: 124057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4383.754080] LustreError: 124057:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [ 4384.074790] Lustre: server umount lustre-MDT0000 complete [ 4390.266393] LustreError: 133002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787578365 with bad export cookie 7672346878301250683 [ 4390.274284] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4393.960819] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4393.983860] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4393.992439] Lustre: Skipped 2 previous similar messages [ 4394.024228] Lustre: Skipped 1 previous similar message [ 4397.286439] Lustre: server umount lustre-MDT0001 complete [ 4409.569861] Lustre: server umount lustre-OST0000 complete [ 4428.255096] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4428.648185] Lustre: server umount lustre-OST0001 complete [ 4439.209303] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 4450.376245] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4492.660603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4492.921369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4492.967620] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4493.043963] Lustre: lustre-MDT0000: new disk, initializing [ 4493.133354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4493.145069] Lustre: Skipped 5 previous similar messages [ 4493.177147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4497.746867] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4510.128626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4510.326826] Lustre: 135988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4510.439703] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4510.444343] Lustre: Skipped 1 previous similar message [ 4510.556763] Lustre: lustre-MDT0001: new disk, initializing [ 4510.729798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4510.751347] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4515.457327] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4526.198853] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4532.662560] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4540.281103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4540.690858] Lustre: lustre-OST0000: new disk, initializing [ 4540.694736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4540.714614] Lustre: 137958:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4540.868678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4540.871742] Lustre: Skipped 1 previous similar message [ 4542.285877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4542.298700] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4542.431041] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4549.097086] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4563.507456] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4581.665274] Lustre: lustre-MDD0000: changelog on [ 4585.079503] Lustre: lustre-MDD0001: changelog on [ 6667.420890] Lustre: lustre-MDD0001: changelog off [ 6669.169841] Lustre: 135996:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6670.126103] Lustre: lustre-MDD0000: changelog off [ 6677.482570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6677.496024] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6677.499045] LustreError: Skipped 1 previous similar message [ 6677.500244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6677.523274] Lustre: Skipped 1 previous similar message [ 6677.527260] Lustre: Skipped 2 previous similar messages [ 6682.603669] LustreError: 137944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6682.637794] LustreError: 137944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6682.767135] Lustre: server umount lustre-OST0000 complete [ 6686.689602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6686.698531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6686.709645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6686.715040] Lustre: Skipped 1 previous similar message [ 6687.713743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6687.715898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6692.833091] LustreError: 135994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6692.953972] Lustre: server umount lustre-MDT0000 complete [ 6696.541925] LustreError: 135981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580671 with bad export cookie 7672346878302030546 [ 6696.544705] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6696.555454] LustreError: 135981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6697.026247] Lustre: server umount lustre-MDT0001 complete [ 6710.514353] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6713.392718] Key type lgssc unregistered [ 6713.621160] LNet: 147123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6713.634842] LNetError: 147123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6714.673688] LNet: Removed LNI 192.168.202.116@tcp [ 6715.905272] Key type .llcrypt unregistered [ 6715.908048] Key type ._llcrypt unregistered [ 6736.912366] Key type ._llcrypt registered [ 6736.914632] Key type .llcrypt registered [ 6737.002024] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 6754.974577] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6756.239212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6756.321289] alg: No test for adler32 (adler32-zlib) [ 6757.488421] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 6757.768546] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6759.415191] Key type lgssc registered [ 6760.378779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6798.814198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6798.849135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6800.366169] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6800.390387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6800.487722] Lustre: lustre-MDT0000: new disk, initializing [ 6800.579388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6800.602720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6805.634035] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6817.996637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6818.159078] Lustre: 150327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6818.217922] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6818.221926] Lustre: Skipped 1 previous similar message [ 6818.297831] Lustre: lustre-MDT0001: new disk, initializing [ 6818.393631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6818.433860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6818.442958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6823.942865] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6835.425958] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6842.958506] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6852.187310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6852.361670] Lustre: lustre-OST0000: new disk, initializing [ 6852.372853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6852.386842] Lustre: 152296:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6852.486727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6858.842625] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6860.850143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6860.868088] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6860.948676] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6870.093994] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6876.640423] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_module obdecho/obdecho [ 6878.880759] LustreError: 153300:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6878.897558] LustreError: 153300:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6880.795568] LustreError: 153398:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6895.072234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6895.084395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6895.111919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6896.611207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6896.614032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6899.060223] Lustre: server umount lustre-MDT0000 complete [ 6902.636989] LustreError: 150319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580877 with bad export cookie 13914367759747204687 [ 6902.639994] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6902.647756] LustreError: 150319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6902.933448] Lustre: server umount lustre-MDT0001 complete [ 6916.382093] Lustre: server umount lustre-OST0000 complete [ 6941.702706] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 6953.610387] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6995.150294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6995.455901] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6995.515606] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6995.660593] Lustre: lustre-MDT0000: new disk, initializing [ 6995.792682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6995.835240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7000.841363] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7013.265979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7013.413448] Lustre: 157118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7013.493648] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7013.503994] Lustre: Skipped 1 previous similar message [ 7013.583313] Lustre: lustre-MDT0001: new disk, initializing [ 7013.694577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7013.747436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7013.767090] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7018.869127] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7029.521597] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7035.738931] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7043.668695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7043.951836] Lustre: lustre-OST0000: new disk, initializing [ 7043.955264] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7043.961867] Lustre: 159086:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7044.056263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7051.268284] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7051.833905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7051.848922] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7052.003489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7061.133740] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7066.894320] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7067.079980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7072.493108] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7072.712478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7077.860562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7077.863515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7077.871345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7077.871354] Lustre: Skipped 2 previous similar messages [ 7077.908014] Lustre: Skipped 2 previous similar messages [ 7081.456528] Lustre: server umount lustre-OST0000 complete [ 7088.099773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7088.102188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7088.125852] Lustre: Skipped 1 previous similar message [ 7088.149422] Lustre: Skipped 2 previous similar messages [ 7093.216359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7093.222144] Lustre: Skipped 1 previous similar message [ 7100.383166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7100.777472] Lustre: server umount lustre-MDT0000 complete [ 7103.458935] LustreError: 157130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7103.483893] LustreError: 157130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7104.335597] LustreError: 157922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581079 with bad export cookie 13914367759747205646 [ 7104.351936] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7104.357136] LustreError: 157922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7104.732581] Lustre: server umount lustre-MDT0001 complete [ 7109.534195] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:18:02 (1787581082) [ 7111.816446] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7114.108554] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:18:07 (1787581087) [ 7122.993533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7123.534551] LustreError: 161500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7123.655355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7128.279982] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7129.058979] LustreError: 161501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7133.160983] LustreError: 161500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7136.792749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7143.296793] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7153.752401] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7161.428921] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7169.994461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7170.534459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7170.547985] Lustre: Skipped 1 previous similar message [ 7179.027806] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7188.878114] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7203.305299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7203.309549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7203.328663] Lustre: Skipped 2 previous similar messages [ 7203.350234] Lustre: Skipped 4 previous similar messages [ 7206.261214] Lustre: server umount lustre-MDT0000 complete [ 7208.418413] LustreError: 161500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7208.451162] LustreError: 161500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7211.018990] LustreError: 161481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581186 with bad export cookie 13914367759747206528 [ 7211.028693] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7211.029698] LustreError: 161481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7213.545409] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7216.547313] LustreError: 163547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7216.577629] LustreError: 163547:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7217.682568] Lustre: server umount lustre-MDT0001 complete [ 7226.684788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7240.752401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7241.187846] LustreError: 165349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7241.275937] LustreError: 165344:0:(llog.c:1655:llog_backup()) MGC192.168.202.116@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7241.282071] Lustre: 165344:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.116@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7276.331745] LustreError: 165351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7276.353572] LustreError: 165351:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 7302.111431] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7302.116398] Lustre: 165344:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.116@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7341.553668] LustreError: 165350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7341.582271] LustreError: 165350:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 41 previous similar messages [ 7343.110692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7348.207113] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7349.560468] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7359.814811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7371.755522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7371.760898] LustreError: Skipped 1 previous similar message [ 7382.002275] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc119c3c189e166d9 [ 7382.014860] Lustre: MGC192.168.202.116@tcp: Connection restored to 0@lo (at 0@lo) [ 7387.651684] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7387.997970] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7399.284953] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7401.430428] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7408.782650] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7411.640304] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7438.827920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7438.837556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7438.846360] Lustre: Skipped 1 previous similar message [ 7438.854171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7438.861035] Lustre: Skipped 6 previous similar messages [ 7440.524175] Lustre: server umount lustre-MDT0000 complete [ 7444.809461] LustreError: 161480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581419 with bad export cookie 13914367759747213017 [ 7444.812750] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7444.826000] LustreError: 161480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7445.274032] Lustre: server umount lustre-MDT0001 complete [ 7461.024902] Lustre: server umount lustre-OST0000 complete [ 7487.016489] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 7497.973931] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7535.955689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7536.145280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7536.184433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7536.241498] Lustre: lustre-MDT0000: new disk, initializing [ 7536.323327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7536.327563] Lustre: Skipped 1 previous similar message [ 7536.341324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7540.577427] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7550.176894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7550.309752] Lustre: 170452:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7550.544223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7555.852217] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7567.479726] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7576.595355] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7585.578809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7585.949016] Lustre: lustre-OST0000: new disk, initializing [ 7585.951301] Lustre: Skipped 1 previous similar message [ 7585.954459] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7585.958400] Lustre: Skipped 2 previous similar messages [ 7585.964781] Lustre: 172425:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7592.864040] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7593.059068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7593.070816] Lustre: Skipped 1 previous similar message [ 7593.083203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7593.152119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7605.106778] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7612.234659] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7612.553079] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7618.375723] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7618.629720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7624.162502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7624.164823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7624.167640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7624.177936] Lustre: Skipped 2 previous similar messages [ 7627.413240] Lustre: server umount lustre-OST0000 complete [ 7632.352225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7632.358994] LustreError: Skipped 1 previous similar message [ 7638.106792] Lustre: server umount lustre-MDT0000 complete [ 7639.524493] LustreError: 171604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7639.539175] LustreError: 171604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 39 previous similar messages [ 7642.111280] LustreError: 170447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581617 with bad export cookie 13914367759747229810 [ 7642.121366] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7642.124056] LustreError: 170447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7642.352351] Lustre: server umount lustre-MDT0001 complete [ 7653.900990] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:27:06 (1787581626) [ 7661.790511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7668.013789] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7677.240973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7684.030568] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7692.491520] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7699.353807] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7706.567164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7714.259815] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7721.533502] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7734.812627] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:28:28 (1787581708) [ 7742.948419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7742.951988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7742.955634] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7742.962753] Lustre: Skipped 3 previous similar messages [ 7742.987318] Lustre: Skipped 5 previous similar messages [ 7745.328082] Lustre: server umount lustre-OST0000 complete [ 7755.694598] Lustre: server umount lustre-MDT0000 complete [ 7760.573798] LustreError: 174820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581735 with bad export cookie 13914367759747230699 [ 7760.582697] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7760.595787] LustreError: 174820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7760.951526] Lustre: server umount lustre-MDT0001 complete [ 7776.203266] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7779.084483] Key type lgssc unregistered [ 7779.728094] LNet: 178570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7779.774090] LNetError: 178570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7779.818067] LNet: Removed LNI 192.168.202.116@tcp [ 7781.268777] Key type .llcrypt unregistered [ 7781.270860] Key type ._llcrypt unregistered [ 7808.390853] Key type ._llcrypt registered [ 7808.394271] Key type .llcrypt registered [ 7808.521794] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7810.396931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7810.472475] alg: No test for adler32 (adler32-zlib) [ 7811.673344] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 7812.077638] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 7813.777065] Key type lgssc registered [ 7815.266624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7825.922864] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7825.942441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7841.631867] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7846.753312] LustreError: 179830:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 7851.871618] LustreError: 179830:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7851.880316] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7851.887635] LustreError: MGC192.168.202.116@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7851.895988] LustreError: 179830:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7851.907942] LustreError: 179830:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7851.921593] LustreError: 179830:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7851.960939] Lustre: server umount lustre-OST0000 complete [ 7851.965625] LustreError: 179830:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7860.929402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7862.372163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7867.378774] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7872.489691] LustreError: 180328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7872.510223] LustreError: 180328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7876.811644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7877.237436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7882.706077] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7894.390612] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7903.628532] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7912.178023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7912.427550] LustreError: 182147:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7912.453660] Lustre: 182147:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.116@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7912.784740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7921.747314] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7930.246772] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7933.409464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7933.421436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7933.438758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7937.507415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7937.514787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7937.531213] Lustre: Skipped 1 previous similar message [ 7938.932778] Lustre: server umount lustre-MDT0000 complete [ 7942.635723] LustreError: 180329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7942.677944] LustreError: 180329:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7942.823210] LustreError: 180309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581917 with bad export cookie 3073814055401678603 [ 7942.837711] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7942.841764] LustreError: 180309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7943.220691] Lustre: server umount lustre-MDT0001 complete [ 7958.223766] Lustre: server umount lustre-OST0000 complete [ 7969.574424] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 7984.868704] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8027.313288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8027.580175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8027.607415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8027.680122] Lustre: lustre-MDT0000: new disk, initializing [ 8027.753281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8027.769551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8033.773703] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8046.894207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8047.042990] Lustre: 185695:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8047.163894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8047.168988] Lustre: Skipped 1 previous similar message [ 8047.286463] Lustre: lustre-MDT0001: new disk, initializing [ 8047.413129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8047.480558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8047.497551] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8053.102797] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8063.495907] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8070.207680] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8078.316337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8078.559610] Lustre: lustre-OST0000: new disk, initializing [ 8078.563561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8078.569580] Lustre: 187664:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8078.667863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8086.040268] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8088.156910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8088.175793] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8088.344126] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8097.686139] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8103.283822] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8103.635610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8109.131365] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8109.302912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8114.145295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8114.147053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8114.162545] LustreError: Skipped 1 previous similar message [ 8114.163655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8114.163659] Lustre: Skipped 2 previous similar messages [ 8114.203036] Lustre: Skipped 1 previous similar message [ 8117.061974] Lustre: server umount lustre-OST0000 complete [ 8124.399402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8124.409065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8124.409365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8124.418162] Lustre: Skipped 1 previous similar message [ 8124.444961] Lustre: Skipped 2 previous similar messages [ 8129.507580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8129.517236] Lustre: Skipped 1 previous similar message [ 8135.137125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8135.517896] Lustre: server umount lustre-MDT0000 complete [ 8139.538585] LustreError: 185687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582114 with bad export cookie 3073814055401679324 [ 8139.541425] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8139.552438] LustreError: 185687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8140.043483] Lustre: server umount lustre-MDT0001 complete [ 8150.834475] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:35:24 (1787582124) [ 8171.575637] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8174.937259] Key type lgssc unregistered [ 8175.274212] LNet: 190615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8175.279315] LNetError: 190615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8176.296683] LNet: Removed LNI 192.168.202.116@tcp [ 8177.681585] Key type .llcrypt unregistered [ 8177.683381] Key type ._llcrypt unregistered [ 8201.515165] Key type ._llcrypt registered [ 8201.517118] Key type .llcrypt registered [ 8201.772684] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8203.613130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8203.699240] alg: No test for adler32 (adler32-zlib) [ 8204.987938] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 8205.311915] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 8207.007373] Key type lgssc registered [ 8208.432827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8217.247943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8217.284378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8219.072675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8224.480239] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8229.352313] LustreError: 191833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8229.373134] LustreError: 191833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8234.470890] LustreError: 191834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8235.015425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8235.433134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8240.466349] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8248.627864] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8255.192265] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8263.451065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8263.614543] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8263.619665] LustreError: 193701:0:(llog.c:1696:llog_backup()) MGC192.168.202.116@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8263.625261] Lustre: 193701:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.116@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8263.711255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8271.111798] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8279.760363] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8285.158414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8285.171564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8285.172891] Lustre: Skipped 2 previous similar messages [ 8285.197820] Lustre: Skipped 2 previous similar messages [ 8287.976834] Lustre: server umount lustre-MDT0000 complete [ 8290.280046] LustreError: 192565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8290.303850] LustreError: 192565:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8291.489744] LustreError: 191816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582266 with bad export cookie 1255812067491650839 [ 8291.496333] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8291.510728] LustreError: 191816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8291.920481] Lustre: server umount lustre-MDT0001 complete [ 8306.646621] Lustre: server umount lustre-OST0000 complete [ 8317.926647] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 8333.945300] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8378.991588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8379.266216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8379.296604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8379.395793] Lustre: lustre-MDT0000: new disk, initializing [ 8379.531096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8379.555796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8384.606432] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8396.541350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8396.678902] Lustre: 197248:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8396.746208] LustreError: 197259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8396.759886] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8396.770928] Lustre: Skipped 1 previous similar message [ 8396.777916] LustreError: 198030:0:(llog.c:1696:llog_backup()) MGC192.168.202.116@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8396.784416] LustreError: 198030:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8396.791832] Lustre: 198030:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.116@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8396.810978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8396.815406] Lustre: Skipped 1 previous similar message [ 8396.900558] Lustre: lustre-MDT0001: new disk, initializing [ 8397.090963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8397.153674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8397.165081] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8402.007770] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8411.849136] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8417.879268] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8424.904142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8425.127622] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8425.130440] Lustre: Skipped 1 previous similar message [ 8425.133714] LustreError: 199191:0:(llog.c:1696:llog_backup()) MGC192.168.202.116@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8425.139676] LustreError: 199191:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8425.145551] Lustre: 199191:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.116@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8425.152631] Lustre: 199191:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8425.171935] Lustre: lustre-OST0000: new disk, initializing [ 8425.175448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8425.183684] Lustre: 199215:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8425.265537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8432.058740] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8434.825344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8434.833428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8435.013288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8442.864367] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8448.938994] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8449.198572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8454.582675] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8454.770698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8460.768389] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8460.779649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8460.783744] LustreError: Skipped 1 previous similar message [ 8460.786101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8460.801596] Lustre: Skipped 1 previous similar message [ 8460.821300] Lustre: Skipped 1 previous similar message [ 8462.596854] Lustre: server umount lustre-OST0000 complete [ 8468.962408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8468.975540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8469.025571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8471.012408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8471.027196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8471.029604] Lustre: Skipped 1 previous similar message [ 8472.873173] Lustre: server umount lustre-MDT0000 complete [ 8476.130555] LustreError: 197259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8476.146855] LustreError: 197259:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8476.862405] LustreError: 199214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582451 with bad export cookie 1255812067491651560 [ 8476.870175] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8476.874289] LustreError: 199214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8477.376806] Lustre: server umount lustre-MDT0001 complete [ 8487.827584] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:41:01 (1787582461) [ 8510.831952] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8522.637221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8523.104340] LustreError: 202367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8523.232749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8528.099860] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8537.659875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8544.464923] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8548.367145] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8557.736760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8558.160675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8558.164948] Lustre: Skipped 1 previous similar message [ 8564.980861] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8574.834742] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8575.233029] Lustre: lustre-OST0001: new disk, initializing [ 8575.267834] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8575.306712] Lustre: 204676:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8582.278919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8582.289201] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8582.454585] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8583.840917] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8597.144621] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8606.235985] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8608.768415] Lustre: DEBUG MARKER: ADD OST3 [ 8615.170025] loop: module loaded [ 8615.243783] loop0: detected capacity change from 0 to 204800000 [ 8615.474034] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8615.481671] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8615.492955] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8623.112673] loop0: detected capacity change from 0 to 204800000 [ 8623.511916] Lustre: DEBUG MARKER: STOP OST3 [ 8629.591217] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8629.990120] Lustre: lustre-OST0003: new disk, initializing [ 8629.992863] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8630.010767] Lustre: 206712:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8630.181102] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8630.184180] Lustre: Skipped 1 previous similar message [ 8631.304888] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8631.602393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8631.635852] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8631.716458] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8633.312244] LustreError: 206814:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8633.519114] Lustre: Failing over lustre-OST0003 [ 8633.833769] Lustre: server umount lustre-OST0003 complete [ 8636.909088] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8636.910680] LustreError: 204370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8636.924718] Lustre: Skipped 1 previous similar message [ 8636.924952] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8636.985772] LustreError: 204370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8637.849643] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8647.700798] Lustre: DEBUG MARKER: START OST3 again [ 8655.280746] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8655.655796] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8657.285980] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8657.709983] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8657.718228] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8662.767360] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8686.140215] Lustre: Failing over lustre-OST0003 [ 8686.234702] Lustre: server umount lustre-OST0003 complete [ 8686.569123] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8686.578618] LustreError: 204370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8686.581192] Lustre: Skipped 1 previous similar message [ 8686.604684] LustreError: 204370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 8707.046245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8707.049200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8707.050398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8707.061146] Lustre: Skipped 1 previous similar message [ 8711.824246] Lustre: server umount lustre-MDT0000 complete [ 8716.447417] LustreError: 202348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582691 with bad export cookie 1255812067491652442 [ 8716.453472] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8716.467307] LustreError: 202348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8717.292295] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8717.308495] Lustre: Skipped 3 previous similar messages [ 8723.021896] Lustre: server umount lustre-MDT0001 complete [ 8727.337801] Lustre: server umount lustre-OST0000 complete [ 8731.484444] Lustre: server umount lustre-OST0001 complete [ 8740.634848] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 8751.014498] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8797.543740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8797.865669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8797.896855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8798.087584] Lustre: lustre-MDT0000: new disk, initializing [ 8798.184322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8798.191261] Lustre: Skipped 1 previous similar message [ 8798.221074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8804.213602] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8814.730924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8814.833902] Lustre: 212137:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8814.953814] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8814.963880] Lustre: Skipped 1 previous similar message [ 8815.106266] Lustre: lustre-MDT0001: new disk, initializing [ 8815.279082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8815.291421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8819.542289] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8829.132681] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8835.755184] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8844.273040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8844.748772] Lustre: 214109:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8846.272636] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8846.450674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8852.398127] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8861.931469] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8866.406507] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8866.555567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8871.111690] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8871.341952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8877.037785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8877.038275] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8877.042310] LustreError: Skipped 1 previous similar message [ 8877.046128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8877.052723] Lustre: Skipped 4 previous similar messages [ 8877.055314] Lustre: Skipped 3 previous similar messages [ 8879.448828] Lustre: server umount lustre-OST0000 complete [ 8886.753169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8889.253761] Lustre: server umount lustre-MDT0000 complete [ 8892.394761] LustreError: 214004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8892.417613] LustreError: 214004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 8893.008873] LustreError: 212131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582868 with bad export cookie 1255812067491654738 [ 8893.011048] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8893.021823] LustreError: 212131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8893.384828] Lustre: server umount lustre-MDT0001 complete [ 8898.224873] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 10:47:51 (1787582871) [ 8918.051695] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 8930.679533] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8968.664559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8968.948171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8969.023559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8969.032427] Lustre: Skipped 1 previous similar message [ 8969.126332] Lustre: lustre-MDT0000: new disk, initializing [ 8969.128920] Lustre: Skipped 1 previous similar message [ 8969.205858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8969.212543] Lustre: Skipped 1 previous similar message [ 8974.336480] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8985.464044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8985.628584] Lustre: 218715:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8985.992425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8991.320464] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9002.168503] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9008.213605] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9016.672582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9017.052411] Lustre: 220688:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9018.813958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9018.965484] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9024.308468] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9034.865188] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9041.942561] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9042.198203] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9047.939126] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9048.152315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9054.692280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9054.693644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9054.700318] LustreError: Skipped 1 previous similar message [ 9054.712093] Lustre: Skipped 2 previous similar messages [ 9054.713880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9054.713887] Lustre: Skipped 4 previous similar messages [ 9056.067670] Lustre: server umount lustre-OST0000 complete [ 9074.144940] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9074.507738] Lustre: server umount lustre-MDT0000 complete [ 9078.833615] LustreError: 219512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583053 with bad export cookie 1255812067491655620 [ 9078.836702] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9078.847440] LustreError: 219512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9079.214648] Lustre: server umount lustre-MDT0001 complete [ 9088.088609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9088.681601] LustreError: 222905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9088.765525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9088.772901] Lustre: Skipped 5 previous similar messages [ 9093.630320] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9103.050538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9107.912409] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9116.760927] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9123.143565] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9130.114454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9137.980981] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9146.367469] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9159.142102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9159.147058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9159.161229] Lustre: Skipped 9 previous similar messages [ 9162.662201] Lustre: server umount lustre-OST0000 complete [ 9175.603767] Lustre: server umount lustre-MDT0000 complete [ 9180.900707] LustreError: 223641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583155 with bad export cookie 1255812067491656495 [ 9180.913148] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9180.919554] LustreError: 223641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9181.755699] Lustre: server umount lustre-MDT0001 complete [ 9285.784207] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 9288.713788] Key type lgssc unregistered [ 9289.018519] LNet: 226576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9289.025392] LNetError: 226576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9289.041725] LNet: Removed LNI 192.168.202.116@tcp [ 9289.788219] Key type .llcrypt unregistered [ 9289.789709] Key type ._llcrypt unregistered [ 9314.146938] Key type ._llcrypt registered [ 9314.149565] Key type .llcrypt registered [ 9314.260674] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 9332.065110] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 9333.273706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9333.329917] alg: No test for adler32 (adler32-zlib) [ 9334.553612] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 9334.799650] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 9336.448895] Key type lgssc registered [ 9337.800927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9375.593560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9375.633637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9376.989390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9377.037682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9377.175082] Lustre: lustre-MDT0000: new disk, initializing [ 9377.351131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9377.414146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9382.217496] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9395.073105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9395.254330] Lustre: 229926:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9395.322290] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9395.330680] Lustre: Skipped 1 previous similar message [ 9395.407353] Lustre: lustre-MDT0001: new disk, initializing [ 9395.517467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9395.601906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9395.625075] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9402.392174] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9412.537848] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9418.231805] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9425.725045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9426.133264] Lustre: lustre-OST0000: new disk, initializing [ 9426.147405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9426.159080] Lustre: 231898:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9426.299321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9433.537775] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9435.210178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9435.223328] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9435.341974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9445.270602] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9453.015453] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9453.364410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9461.282384] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9461.699674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9466.336790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9466.345295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9466.355108] LustreError: Skipped 1 previous similar message [ 9466.357984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9466.416152] Lustre: Skipped 1 previous similar message [ 9470.843544] Lustre: server umount lustre-OST0000 complete [ 9476.583375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9476.599233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9476.614570] Lustre: Skipped 2 previous similar messages [ 9481.696920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9481.710912] Lustre: Skipped 1 previous similar message [ 9486.816857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9486.829092] Lustre: Skipped 1 previous similar message [ 9490.399233] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9490.672919] Lustre: server umount lustre-MDT0000 complete [ 9494.746864] LustreError: 231897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583469 with bad export cookie 5940286855579699523 [ 9494.753295] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9494.756801] LustreError: 231897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9495.026274] Lustre: server umount lustre-MDT0001 complete [ 9516.416578] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 9526.845656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9527.314122] LustreError: 234835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9527.344794] LustreError: 234835:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9527.419212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9531.993261] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9532.896620] LustreError: 234836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9537.001134] LustreError: 234835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9540.941105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9541.267954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9546.058673] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9549.224126] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9557.339935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9557.785736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9564.231605] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9574.664905] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9574.892723] Lustre: lustre-OST0001: new disk, initializing [ 9574.901696] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9574.910599] Lustre: 237145:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9575.086927] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9582.333569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9582.353983] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9582.550505] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9583.508514] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9596.455757] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9604.236648] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9617.667781] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 10:59:51 (1787583591) [ 9623.521731] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9623.530300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9623.535336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9623.536687] Lustre: Skipped 3 previous similar messages [ 9626.508069] Lustre: server umount lustre-MDT0000 complete [ 9628.646858] LustreError: 234832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9628.682874] LustreError: 234832:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9630.533736] LustreError: 234817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583605 with bad export cookie 5940286855579700405 [ 9630.554801] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9630.557155] LustreError: 234817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9631.237868] Lustre: server umount lustre-MDT0001 complete [ 9646.758990] Lustre: server umount lustre-OST0000 complete [ 9649.702229] Lustre: 228150:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787583608/real 1787583608] req@ffff946bbd9ba680 x1874416955935872/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787583624 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9649.752170] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9649.779991] Lustre: Skipped 1 previous similar message [ 9650.978217] Lustre: server umount lustre-OST0001 complete [ 9660.944515] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 9675.752504] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 9720.440762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9720.759480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9720.809589] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9720.912663] Lustre: lustre-MDT0000: new disk, initializing [ 9721.039571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9721.057595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9725.706113] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9739.188405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9739.319248] Lustre: 241101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9739.394456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9739.399037] Lustre: Skipped 1 previous similar message [ 9739.539469] Lustre: lustre-MDT0001: new disk, initializing [ 9739.723083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9739.738456] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9744.418580] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9754.490554] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9760.882152] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9768.367858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9768.659455] Lustre: lustre-OST0000: new disk, initializing [ 9768.661576] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9768.665139] Lustre: 243069:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9770.663765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9770.678886] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9770.789959] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9775.163623] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9788.004545] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9794.914207] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9795.176188] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9802.349868] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9802.785956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9806.821532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9806.840562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9806.845422] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9806.856060] Lustre: Skipped 4 previous similar messages [ 9811.388411] Lustre: server umount lustre-OST0000 complete [ 9816.545065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9816.553414] LustreError: Skipped 1 previous similar message [ 9816.561448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9816.580373] Lustre: Skipped 1 previous similar message [ 9822.180967] LustreError: 241106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9822.207032] LustreError: 241106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9822.225320] Lustre: server umount lustre-MDT0000 complete [ 9827.119018] LustreError: 241092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583802 with bad export cookie 5940286855579702540 [ 9827.125329] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9827.135961] LustreError: 241092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9827.598645] Lustre: server umount lustre-MDT0001 complete [ 9837.250317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9837.531183] LustreError: 245288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9837.547809] LustreError: 245288:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9837.588883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9837.595811] Lustre: Skipped 2 previous similar messages [ 9843.092484] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9853.943083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9859.081452] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9867.923354] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9874.559686] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9882.137041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9889.739332] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9898.040175] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10042.338642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10042.355849] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10042.368029] Lustre: Skipped 1 previous similar message [10042.379614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10042.394565] Lustre: Skipped 3 previous similar messages [10048.238822] Lustre: server umount lustre-OST0000 complete [10054.113652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10054.122126] LustreError: Skipped 1 previous similar message [10059.222981] Lustre: server umount lustre-MDT0000 complete [10060.770286] LustreError: 246292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10060.790194] LustreError: 246292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10063.673824] LustreError: 245269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584038 with bad export cookie 5940286855579703422 [10063.679780] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10063.689305] LustreError: 245269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10064.106821] Lustre: server umount lustre-MDT0001 complete [10167.345686] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [10170.599460] Key type lgssc unregistered [10170.920780] LNet: 248907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10170.929626] LNetError: 248907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10170.954128] LNet: Removed LNI 192.168.202.116@tcp [10171.871160] Key type .llcrypt unregistered [10171.873386] Key type ._llcrypt unregistered [10198.897290] Key type ._llcrypt registered [10198.899249] Key type .llcrypt registered [10198.984642] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [10214.980763] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [10216.140994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10216.202676] alg: No test for adler32 (adler32-zlib) [10217.192795] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [10217.411118] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [10219.087183] Key type lgssc registered [10220.145591] Lustre: Echo OBD driver; http://www.lustre.org/ [10262.532734] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [10273.398835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10273.423192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10274.649864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10274.703067] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10274.852639] Lustre: lustre-MDT0000: new disk, initializing [10274.936578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10274.949617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10279.946837] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10291.440926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10291.599539] Lustre: 252974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10291.666489] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10291.670113] Lustre: Skipped 1 previous similar message [10291.792715] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10291.801878] Lustre: lustre-MDT0001: new disk, initializing [10291.918891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10291.976615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10291.996913] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10296.500155] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10301.937321] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10309.242398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10309.629944] Lustre: lustre-OST0000: new disk, initializing [10309.640064] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10309.655927] Lustre: 254605:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10309.830786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10315.261725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10315.274708] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10315.353199] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10317.057491] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10332.332343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10332.712793] Lustre: lustre-OST0001: new disk, initializing [10332.717458] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10332.725492] Lustre: 255479:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10333.158337] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10342.100245] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10342.104880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10342.130369] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10342.228261] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10355.208733] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10363.383722] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10378.218947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10378.231381] Lustre: Skipped 2 previous similar messages [10378.233732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10378.247403] Lustre: Skipped 3 previous similar messages [10381.477286] Lustre: server umount lustre-MDT0000 complete [10385.050670] LustreError: 254236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584360 with bad export cookie 16464710722520134902 [10385.055855] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10385.067709] LustreError: 254236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10385.336871] Lustre: server umount lustre-MDT0001 complete [10400.024647] Lustre: server umount lustre-OST0000 complete [10414.997417] Lustre: server umount lustre-OST0001 complete [10428.049841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10438.007515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10446.752917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10456.549644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10477.190763] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [10490.674650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10490.701066] Lustre: lustre-MDT0000: reset Object Index mappings [10491.222986] LustreError: 260490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10491.238613] LustreError: 260490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10491.333588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10495.399929] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10496.483642] LustreError: 260491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10501.604268] LustreError: 260490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10505.465020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10505.483432] Lustre: lustre-MDT0001: reset Object Index mappings [10506.082133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10512.136689] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10514.929993] Lustre: 261635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10521.781903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10522.184695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10528.139980] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10530.429338] LustreError: 261989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10530.453848] LustreError: 261989:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10535.914300] LustreError: 261989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10537.465222] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10538.873693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10538.910630] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10544.481586] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10553.188171] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10557.663754] Lustre: 263506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10562.535408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10562.538678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10562.565822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10564.593518] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10564.602284] Lustre: Skipped 2 previous similar messages [10564.615119] LustreError: 260486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10564.627637] Lustre: server umount lustre-MDT0000 complete [10564.633530] LustreError: 260486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10568.003754] LustreError: 260472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584543 with bad export cookie 16464710722520141685 [10568.004907] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10568.018084] LustreError: 260472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10568.324665] Lustre: server umount lustre-MDT0001 complete [10582.595667] Lustre: server umount lustre-OST0000 complete [10585.897147] Lustre: 250482:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787584544/real 1787584544] req@ffff946bbb958a80 x1874417881842816/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787584560 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10585.945406] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10586.763180] Lustre: server umount lustre-OST0001 complete [10606.395732] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:16:20 (1787584580) [10622.668753] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [10635.311814] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [10673.793692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10674.041965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10674.077738] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10674.164317] Lustre: lustre-MDT0000: new disk, initializing [10674.302370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10674.311741] Lustre: Skipped 1 previous similar message [10674.324730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10679.314952] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10691.113030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10691.263557] Lustre: 267761:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10691.321674] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10691.331816] Lustre: Skipped 1 previous similar message [10691.447648] Lustre: lustre-MDT0001: new disk, initializing [10691.690801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10691.698981] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10696.947654] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10707.365762] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10714.895489] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10723.041812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10723.566547] Lustre: lustre-OST0000: new disk, initializing [10723.579686] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10723.589216] Lustre: 269728:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10724.985882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10725.024944] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10725.199593] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10730.823099] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10741.915956] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10748.547912] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10748.842950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10754.844799] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10755.279221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10761.198673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10761.210397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10761.212210] LustreError: Skipped 1 previous similar message [10761.212842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10761.243835] Lustre: Skipped 1 previous similar message [10764.572766] Lustre: server umount lustre-OST0000 complete [10771.428815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10771.462899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10771.465275] Lustre: Skipped 1 previous similar message [10775.764622] Lustre: server umount lustre-MDT0000 complete [10776.548143] LustreError: 268257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10776.580126] LustreError: 268257:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10779.555526] LustreError: 268555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584754 with bad export cookie 16464710722520143778 [10779.564997] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10780.098706] Lustre: server umount lustre-MDT0001 complete [10802.988683] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [10814.524039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10814.890800] LustreError: 272667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10814.982828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10814.992769] Lustre: Skipped 2 previous similar messages [10820.119318] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10830.453474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10835.389898] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10838.947797] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10846.269604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10853.631949] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10863.663596] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10863.819697] Lustre: lustre-OST0001: new disk, initializing [10863.824773] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10863.831066] Lustre: 274977:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10865.719556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10865.740419] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10865.873896] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10871.438291] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10884.032741] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10888.761888] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10895.844270] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10895.900115] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10907.120652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10907.130431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10907.147977] Lustre: Skipped 3 previous similar messages [10907.160827] Lustre: Skipped 3 previous similar messages [10909.344474] Lustre: server umount lustre-MDT0000 complete [10912.231200] LustreError: 272664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10912.269788] LustreError: 272664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10913.567093] LustreError: 272648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584888 with bad export cookie 16464710722520144667 [10913.581747] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10914.061658] Lustre: server umount lustre-MDT0001 complete [10929.800701] Lustre: server umount lustre-OST0000 complete [10933.727341] Lustre: 250482:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787584892/real 1787584892] req@ffff946a82f24700 x1874417881958272/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787584908 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10934.751682] Lustre: 250484:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787584893/real 1787584893] req@ffff946bc1707480 x1874417881958528/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787584909 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10935.640201] Lustre: server umount lustre-OST0001 complete [10955.146818] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [10966.285500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10967.076367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10967.083901] Lustre: Skipped 3 previous similar messages [10972.813687] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10983.064642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10988.984504] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10992.223800] Lustre: 279177:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10999.212626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11005.936743] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11012.802626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11015.952801] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11021.373689] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11022.849813] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11031.119252] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11035.676230] Lustre: 281052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11042.738065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11042.849105] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11053.409636] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:23:46 (1787585026) [11070.432787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11070.443869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11070.467922] Lustre: Skipped 2 previous similar messages [11070.474552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11074.803475] Lustre: server umount lustre-MDT0000 complete [11077.613869] LustreError: 278032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11077.640402] LustreError: 278032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [11078.750257] LustreError: 278015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585053 with bad export cookie 16464710722520151219 [11078.766133] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11078.775098] LustreError: 278015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11085.401510] Lustre: server umount lustre-MDT0001 complete [11104.224164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11104.392326] Lustre: server umount lustre-OST0000 complete [11108.644933] Lustre: server umount lustre-OST0001 complete [11117.129533] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [11131.356474] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [11188.717864] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [11203.458745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11204.125399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11204.164883] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11204.302183] Lustre: lustre-MDT0000: new disk, initializing [11204.558084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11211.268350] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11224.415678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11224.512727] Lustre: 285325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11224.889342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11224.898621] Lustre: Skipped 4 previous similar messages [11224.954933] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11231.225204] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11237.241304] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11246.830964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11247.272643] Lustre: 286961:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11249.039336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11254.540275] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11266.643678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11266.851980] Lustre: 287833:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11268.545451] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11268.553240] Lustre: Skipped 1 previous similar message [11268.681811] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11274.791817] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11287.554959] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11309.984327] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 11:28:03 (1787585283) [11312.096316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11312.106740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11312.125587] Lustre: Skipped 5 previous similar messages [11312.137073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11312.149762] Lustre: Skipped 8 previous similar messages [11314.508369] Lustre: server umount lustre-MDT0000 complete [11317.956183] LustreError: 285318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585292 with bad export cookie 16464710722520158898 [11317.959401] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11317.980552] LustreError: 285318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11318.280103] Lustre: server umount lustre-MDT0001 complete [11332.960984] Lustre: server umount lustre-OST0000 complete [11335.903122] Lustre: 250485:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787585294/real 1787585294] req@ffff946bbb7f5500 x1874417882116992/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787585310 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11336.639676] Lustre: server umount lustre-OST0001 complete [11347.279833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11347.929587] LustreError: 290079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11347.957341] LustreError: 290079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [11353.968250] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11363.167685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11368.038876] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11376.436539] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11382.897968] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11389.640410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11396.048465] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11404.852880] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11427.045072] loop1: detected capacity change from 0 to 204800000 [11427.365304] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11427.374049] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11427.399020] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11433.926529] loop1: detected capacity change from 0 to 204800000 [11440.009645] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11440.129336] LustreError: 290066:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs3886: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11440.137808] LustreError: 290066:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs3886 [11440.142742] LustreError: 290066:0:(mgs_handler.c:517:mgs_target_reg()) fs3886-MDT0001: Error checking target index: rc = -13 [11440.156369] LustreError: fs3886-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11440.171758] LustreError: 293800:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11440.186319] LustreError: 293800:0:(tgt_mount.c:2132:server_put_super()) no obd fs3886-MDT0001 [11440.197527] LustreError: 293800:0:(tgt_mount.c:116:server_deregister_mount()) fs3886-MDT0001 not registered [11440.234973] Lustre: server umount fs3886-MDT0001 complete [11440.237654] LustreError: 293800:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11449.567298] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11449.635652] Lustre: Setting parameter fs3886-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs3886-MDT0001 [11449.642580] Lustre: Skipped 1 previous similar message [11449.665946] Lustre: srv-fs3886-MDT0001: No data found on store. Initialize space. [11449.668323] Lustre: Skipped 4 previous similar messages [11449.730773] Lustre: fs3886-MDT0001: new disk, initializing [11449.739492] Lustre: Skipped 3 previous similar messages [11449.899767] Lustre: srv-fs3886-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11449.912795] Lustre: 294266:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3886-MDT0001-mdtlov: can't create new llog: rc = -115 [11449.919688] LustreError: 294266:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3886-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11454.937983] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11462.299570] Lustre: server umount fs3886-MDT0001 complete [11471.331816] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11478.002986] loop1: detected capacity change from 0 to 204800000 [11484.447979] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11484.804903] Lustre: 295836:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3886-MDT0001-mdtlov: can't create new llog: rc = -115 [11484.816411] LustreError: 295836:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3886-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11489.579071] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11491.820122] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11495.190761] Lustre: server umount fs3886-MDT0001 complete [11506.658376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11506.672133] Lustre: Skipped 4 previous similar messages [11506.680405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11511.526508] Lustre: server umount lustre-MDT0000 complete [11520.514398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11520.674894] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11525.711481] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11526.127368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11533.341370] loop1: detected capacity change from 0 to 204800000 [11539.329104] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11539.580343] Lustre: 298282:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3886-MDT0001-mdtlov: can't create new llog: rc = -115 [11539.585091] LustreError: 298282:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3886-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11543.765308] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11548.755903] Lustre: server umount fs3886-MDT0001 complete [11572.212638] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11572.219074] LustreError: Skipped 1 previous similar message [11578.387266] Lustre: server umount lustre-OST0000 complete [11596.259507] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11596.665548] Lustre: server umount lustre-MDT0000 complete [11600.212099] LustreError: 290059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585575 with bad export cookie 16464710722520161579 [11600.222433] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11600.227901] LustreError: 290059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11600.570919] Lustre: server umount lustre-MDT0001 complete [11611.559400] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [11614.537717] Key type lgssc unregistered [11614.771805] LNet: 300631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11614.777736] LNetError: 300631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11614.791579] LNet: Removed LNI 192.168.202.116@tcp [11615.805172] Key type .llcrypt unregistered [11615.813071] Key type ._llcrypt unregistered [11639.819778] Key type ._llcrypt registered [11639.827660] Key type .llcrypt registered [11639.938192] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [11641.706971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11641.756855] alg: No test for adler32 (adler32-zlib) [11643.097887] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [11643.589657] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [11645.377974] Key type lgssc registered [11647.129187] Lustre: Echo OBD driver; http://www.lustre.org/ [11656.349785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11656.375101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11658.070854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11663.104356] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11668.458786] LustreError: 301908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11668.480135] LustreError: 301908:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11671.729658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11672.056476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11676.035404] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11685.115930] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11692.162966] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11699.396945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11699.898688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11704.007618] LustreError: 303741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11706.411794] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11709.440955] LustreError: 303964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11714.210488] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11714.531329] LustreError: 303741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11721.711325] LustreError: 303964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11721.743312] LustreError: 303964:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11731.958582] LustreError: 303964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11731.971855] LustreError: 303964:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11732.249862] loop1: detected capacity change from 0 to 204800000 [11732.388414] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11732.395898] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11732.402993] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11737.188222] loop1: detected capacity change from 0 to 204800000 [11744.166752] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11744.487218] Lustre: fs16199-OST0000: new disk, initializing [11744.492344] Lustre: srv-fs16199-OST0000: No data found on store. Initialize space. [11744.500581] Lustre: 305644:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11744.592055] Lustre: fs16199-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11752.429187] LustreError: 303738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11752.462369] LustreError: 303738:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [11754.223372] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11765.098393] Lustre: server umount fs16199-OST0000 complete [11778.256353] loop1: detected capacity change from 0 to 204800000 [11783.154090] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11784.860221] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11785.355726] Lustre: fs16199-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11788.261511] LustreError: 303738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11788.305891] LustreError: 303738:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [11792.464122] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11798.515563] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11802.384629] loop2: detected capacity change from 0 to 204800000 [11802.680261] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11802.692887] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11802.708706] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11809.011060] loop2: detected capacity change from 0 to 204800000 [11816.596601] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11816.686186] LustreError: fs16199-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11816.708950] LustreError: fs16199-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11816.726488] LustreError: 308685:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11816.743188] LustreError: 308685:0:(tgt_mount.c:2132:server_put_super()) no obd fs16199-OST0001 [11816.748848] LustreError: 308685:0:(tgt_mount.c:116:server_deregister_mount()) fs16199-OST0001 not registered [11816.792834] Lustre: server umount fs16199-OST0001 complete [11816.795917] LustreError: 308685:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11829.311296] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11829.586524] Lustre: fs16199-OST0001: new disk, initializing [11829.604925] Lustre: srv-fs16199-OST0001: No data found on store. Initialize space. [11829.629264] Lustre: 309238:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11829.758745] Lustre: fs16199-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11838.203857] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11847.609214] Lustre: server umount fs16199-OST0000 complete [11854.817570] LustreError: 303738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11854.851272] LustreError: 303738:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11858.186944] Lustre: server umount fs16199-OST0001 complete [11872.142329] loop1: detected capacity change from 0 to 204800000 [11879.175335] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11879.419395] Lustre: fs16199-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11886.089799] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11892.822080] loop2: detected capacity change from 0 to 204800000 [11899.102454] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11905.908703] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11915.813100] Lustre: server umount fs16199-OST0000 complete [11925.782735] Lustre: server umount fs16199-OST0001 complete [11936.754332] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11939.810611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11939.821038] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11939.844444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11941.859714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11941.864490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11941.895559] Lustre: Skipped 1 previous similar message [11944.708807] Lustre: server umount lustre-OST0000 complete [11952.103091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11952.105625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11952.128016] Lustre: Skipped 1 previous similar message [11952.153102] Lustre: Skipped 1 previous similar message [11957.217838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11957.226557] Lustre: Skipped 1 previous similar message [11962.339596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11962.349868] Lustre: Skipped 1 previous similar message [11963.359246] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11963.493282] Lustre: server umount lustre-MDT0000 complete [11967.490485] LustreError: 301889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585942 with bad export cookie 9132440799408436515 [11967.506288] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11967.522651] LustreError: 301889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11968.184628] Lustre: server umount lustre-MDT0001 complete [11981.305987] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [11984.662690] Key type lgssc unregistered [11985.002610] LNet: 315167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11985.013619] LNetError: 315167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11985.024239] LNet: Removed LNI 192.168.202.116@tcp [11986.078723] Key type .llcrypt unregistered [11986.084307] Key type ._llcrypt unregistered [12026.059809] Key type ._llcrypt registered [12026.066116] Key type .llcrypt registered [12026.263885] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [12045.101646] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [12047.585360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12047.608641] alg: No test for adler32 (adler32-zlib) [12048.923507] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [12049.259707] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [12050.975175] Key type lgssc registered [12051.968826] Lustre: Echo OBD driver; http://www.lustre.org/ [12095.808082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12095.825402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12097.229559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12097.270879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12097.425164] Lustre: lustre-MDT0000: new disk, initializing [12097.564864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12097.607282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12102.596238] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12115.733793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12115.980439] Lustre: 319033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12116.053552] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12116.060175] Lustre: Skipped 1 previous similar message [12116.184773] Lustre: lustre-MDT0001: new disk, initializing [12116.302493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12116.355740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12116.372994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12121.876165] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12134.396810] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12139.155859] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12148.830987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12149.350892] Lustre: lustre-OST0000: new disk, initializing [12149.355926] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12149.369784] Lustre: 321003:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12149.579423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12156.810168] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12159.027151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12159.035464] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12159.160953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12168.020292] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12170.860917] Lustre: Mounted lustre-client - version 2.17.57_80_g9b8a413 [12172.200135] Lustre: Unmounted lustre-client [12185.060248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12185.060579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12185.064955] LustreError: Skipped 1 previous similar message [12185.065896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12185.082274] Lustre: Skipped 1 previous similar message [12185.101302] Lustre: Skipped 1 previous similar message [12186.081866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12186.097099] Lustre: Skipped 1 previous similar message [12189.969776] Lustre: server umount lustre-OST0000 complete [12196.320669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12196.325437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12196.327726] Lustre: Skipped 1 previous similar message [12196.342712] Lustre: Skipped 1 previous similar message [12200.701331] Lustre: server umount lustre-MDT0000 complete [12204.935839] LustreError: 319026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586179 with bad export cookie 15339653525351192072 [12204.947398] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12205.201790] Lustre: server umount lustre-MDT0001 complete [12217.106461] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [12220.107755] Key type lgssc unregistered [12231.647172] LNet: 1 local NIs in recovery (showing 1): 192.168.202.116@tcp [12231.711679] LNet: 323168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12231.717232] LNetError: 323168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12301.369131] LNet: Removed LNI 192.168.202.116@tcp [12302.732776] Key type .llcrypt unregistered [12302.741402] Key type ._llcrypt unregistered [12326.090439] Key type ._llcrypt registered [12326.092424] Key type .llcrypt registered [12326.208979] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [12342.303236] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [12343.303921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12343.375885] alg: No test for adler32 (adler32-zlib) [12344.525630] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [12344.771877] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [12346.455200] Key type lgssc registered [12347.964109] Lustre: Echo OBD driver; http://www.lustre.org/ [12380.755226] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 11:45:53 (1787586353) [12389.125889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12389.162792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12390.542632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12390.599300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12390.748526] Lustre: lustre-MDT0000: new disk, initializing [12390.902858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12390.936524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12396.488678] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12410.523804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12410.721513] Lustre: 326392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12410.835720] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12410.847160] Lustre: Skipped 1 previous similar message [12411.013959] Lustre: lustre-MDT0001: new disk, initializing [12411.197510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12411.252924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12411.266436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12417.025311] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12427.202864] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12434.855858] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12443.985144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12444.294750] Lustre: lustre-OST0000: new disk, initializing [12444.297033] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12444.303286] Lustre: 328363:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12444.379315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12452.465758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12452.479349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12452.587050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12453.100994] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12463.661410] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12483.556268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12483.559069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12483.559391] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12483.584184] Lustre: Skipped 1 previous similar message [12483.601714] Lustre: Skipped 1 previous similar message [12485.334140] Lustre: server umount lustre-OST0000 complete [12493.281390] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12493.290411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12493.311869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12496.271308] Lustre: server umount lustre-MDT0000 complete [12500.523457] LustreError: 326386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586475 with bad export cookie 5766951482736288143 [12500.532690] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12500.534792] LustreError: 326386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12500.835878] Lustre: server umount lustre-MDT0001 complete [12512.737525] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [12516.359452] Key type lgssc unregistered [12516.574744] LNet: 330273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12516.579866] LNetError: 330273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12516.593327] LNet: Removed LNI 192.168.202.116@tcp [12517.514168] Key type .llcrypt unregistered [12517.520868] Key type ._llcrypt unregistered [12543.973050] Key type ._llcrypt registered [12543.978401] Key type .llcrypt registered [12544.131886] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [12545.075972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12545.091812] alg: No test for adler32 (adler32-zlib) [12546.510353] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [12546.847287] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [12548.679739] Key type lgssc registered [12549.854827] Lustre: Echo OBD driver; http://www.lustre.org/ [12558.404166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12558.416681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12559.968266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12565.599570] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12569.072823] LustreError: 331551:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12569.090220] LustreError: 331551:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12574.177851] LustreError: 331552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12574.661925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12574.962813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12580.738109] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12590.143416] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12597.115363] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12604.476476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12604.966888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12611.041865] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12618.738808] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12628.710312] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12630.811985] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12636.651111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12636.659983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12636.670398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12639.716871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12639.718961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12639.723023] Lustre: Skipped 1 previous similar message [12639.759531] Lustre: Skipped 1 previous similar message [12641.643268] Lustre: server umount lustre-MDT0000 complete [12642.111074] LustreError: 332801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12644.840971] LustreError: 331552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12644.868757] LustreError: 331552:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12645.984159] LustreError: 331531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586621 with bad export cookie 644630313110883312 [12645.988092] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12645.994822] LustreError: 331531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12647.260226] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.16@tcp (stopping) [12647.273465] Lustre: Skipped 2 previous similar messages [12649.953357] LustreError: 334040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12649.954243] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12649.974381] LustreError: 334040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12650.009216] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12652.311174] Lustre: server umount lustre-MDT0001 complete [12656.253661] Lustre: server umount lustre-OST0000 complete [12664.597163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12669.457890] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12672.280423] Lustre: 335732:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.116@tcp [12674.511901] Lustre: server umount MGS complete [12683.106638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12683.509906] LustreError: 336303:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12683.532434] LustreError: 336303:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12683.565621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12687.857594] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12696.387487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12696.622090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12702.170650] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12711.576212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12711.925310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12718.426942] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12727.344823] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12729.399388] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12734.928797] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12736.765455] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12741.602990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12741.608797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12741.626238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12741.634952] Lustre: Skipped 1 previous similar message [12743.651723] Lustre: server umount lustre-OST0000 complete [12749.794784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12749.803842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12759.088728] Lustre: server umount lustre-MDT0000 complete [12763.795064] LustreError: 336285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586738 with bad export cookie 644630313110884152 [12763.815816] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12763.830838] LustreError: 336285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12764.712574] Lustre: server umount lustre-MDT0001 complete [12778.013151] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [12782.080634] Key type lgssc unregistered [12782.606796] LNet: 339741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12782.612262] LNetError: 339741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12782.636634] LNet: Removed LNI 192.168.202.116@tcp [12783.773589] Key type .llcrypt unregistered [12783.775620] Key type ._llcrypt unregistered [12804.366419] Key type ._llcrypt registered [12804.368121] Key type .llcrypt registered [12804.500207] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [12821.818107] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [12823.175805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12823.252495] alg: No test for adler32 (adler32-zlib) [12824.261786] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [12824.448034] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [12826.137464] Key type lgssc registered [12827.303896] Lustre: Echo OBD driver; http://www.lustre.org/ [12864.087637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12864.123747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12865.508912] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12865.637548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12865.749241] Lustre: lustre-MDT0000: new disk, initializing [12865.956631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12865.991783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12871.996867] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12883.524839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12883.695805] Lustre: 342903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12883.747521] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12883.749716] Lustre: Skipped 1 previous similar message [12883.789531] Lustre: lustre-MDT0001: new disk, initializing [12883.869148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12883.908531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12883.921868] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12889.444281] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12899.200476] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12905.231798] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12912.362448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12912.597403] Lustre: lustre-OST0000: new disk, initializing [12912.600750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12912.605673] Lustre: 344870:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12912.689235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12919.520488] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12920.856302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12920.861852] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12921.032465] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12929.606245] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12934.602122] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12934.873362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12939.782041] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12940.030362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12946.413331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12946.420800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12946.459712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12948.285898] Lustre: server umount lustre-OST0000 complete [12955.619268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12955.628839] LustreError: Skipped 1 previous similar message [12955.638444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12955.654023] Lustre: Skipped 1 previous similar message [12955.660266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12955.664363] Lustre: Skipped 1 previous similar message [12957.157098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12957.159380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12958.980594] Lustre: server umount lustre-MDT0000 complete [12962.668414] LustreError: 342896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586937 with bad export cookie 16753735201005914039 [12962.678860] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12962.685585] LustreError: 342896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12962.993986] Lustre: server umount lustre-MDT0001 complete [12972.588602] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 11:55:46 (1787586946) [12988.496846] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [12999.823154] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [13033.706056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13034.078557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13034.097904] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13034.157342] Lustre: lustre-MDT0000: new disk, initializing [13034.213219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13034.224061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13038.537692] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [13049.040946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13049.211783] Lustre: 349636:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13049.271584] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13049.279195] Lustre: Skipped 1 previous similar message [13049.343450] Lustre: lustre-MDT0001: new disk, initializing [13049.429574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13049.464189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13049.485847] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13053.251659] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [13063.060572] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13069.203113] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13076.214128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13076.549779] Lustre: lustre-OST0000: new disk, initializing [13076.557696] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13076.575933] Lustre: 351602:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13076.684135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13083.098168] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [13083.226031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13083.231488] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13083.339096] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13092.562995] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13129.723636] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13129.724607] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13129.727776] LustreError: Skipped 1 previous similar message [13129.729756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13129.729764] Lustre: Skipped 1 previous similar message [13129.750635] Lustre: Skipped 1 previous similar message [13134.825185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13134.835440] Lustre: Skipped 2 previous similar messages [13135.120524] Lustre: server umount lustre-OST0000 complete [13139.937784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13144.660560] Lustre: server umount lustre-MDT0000 complete [13145.067805] LustreError: 349646:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13145.091903] LustreError: 349646:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13148.715265] LustreError: 349626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587123 with bad export cookie 16753735201005914921 [13148.718655] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13148.725697] LustreError: 349626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [13149.306874] Lustre: server umount lustre-MDT0001 complete [13162.182257] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [13164.762155] Key type lgssc unregistered [13164.978831] LNet: 353780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13164.983604] LNetError: 353780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13164.996475] LNet: Removed LNI 192.168.202.116@tcp [13165.833318] Key type .llcrypt unregistered [13165.836166] Key type ._llcrypt unregistered [13263.133812] Key type ._llcrypt registered [13263.135900] Key type .llcrypt registered [13263.329538] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [13264.067630] Key type .llcrypt unregistered [13264.090570] Key type ._llcrypt unregistered [13340.550886] Key type ._llcrypt registered [13340.553086] Key type .llcrypt registered [13340.754429] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [13341.083276] Key type .llcrypt unregistered [13341.085607] Key type ._llcrypt unregistered [13370.155633] Key type ._llcrypt registered [13370.162639] Key type .llcrypt registered [13370.287930] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [13370.570316] Key type .llcrypt unregistered [13370.572081] Key type ._llcrypt unregistered [13405.961777] Key type ._llcrypt registered [13405.963866] Key type .llcrypt registered [13406.112938] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [13406.596388] Key type .llcrypt unregistered [13406.599400] Key type ._llcrypt unregistered [13491.241860] Key type ._llcrypt registered [13491.244653] Key type .llcrypt registered [13491.334454] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [13491.619379] Key type .llcrypt unregistered [13491.621283] Key type ._llcrypt unregistered [13521.783365] Key type ._llcrypt registered [13521.786741] Key type .llcrypt registered [13521.896427] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [13522.226639] Key type .llcrypt unregistered [13522.228398] Key type ._llcrypt unregistered [13553.511179] Key type ._llcrypt registered [13553.517781] Key type .llcrypt registered [13553.666225] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [13554.005320] Key type .llcrypt unregistered [13554.008957] Key type ._llcrypt unregistered [13595.216910] Key type ._llcrypt registered [13595.220087] Key type .llcrypt registered [13595.344649] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [13595.972039] Key type .llcrypt unregistered [13595.981662] Key type ._llcrypt unregistered [13634.161417] Key type ._llcrypt registered [13634.162883] Key type .llcrypt registered [13634.292195] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [13634.705282] Key type .llcrypt unregistered [13634.707461] Key type ._llcrypt unregistered [13671.383797] Key type ._llcrypt registered [13671.385789] Key type .llcrypt registered [13671.493496] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [13671.915811] Key type .llcrypt unregistered [13671.917942] Key type ._llcrypt unregistered [13740.512548] Key type ._llcrypt registered [13740.514562] Key type .llcrypt registered [13740.631565] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [13740.877721] Key type .llcrypt unregistered [13740.879837] Key type ._llcrypt unregistered [13770.265699] Key type ._llcrypt registered [13770.267797] Key type .llcrypt registered [13770.351583] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [13770.582451] Key type .llcrypt unregistered [13770.584347] Key type ._llcrypt unregistered [13803.368343] Key type ._llcrypt registered [13803.370732] Key type .llcrypt registered [13803.572049] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [13803.992486] Key type .llcrypt unregistered [13803.999819] Key type ._llcrypt unregistered [13860.022791] Key type ._llcrypt registered [13860.025273] Key type .llcrypt registered [13860.113241] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [13876.259534] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [13877.256379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13877.379504] alg: No test for adler32 (adler32-zlib) [13878.503796] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [13878.787321] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [13880.455720] Key type lgssc registered [13881.558928] Lustre: Echo OBD driver; http://www.lustre.org/ [13915.536064] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13915.563095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13916.954728] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13917.004595] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13917.073280] Lustre: tst250fs-MDT0000: new disk, initializing [13917.153784] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13917.173384] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13922.029278] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [13932.807564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13932.917808] Lustre: 374213:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13932.969903] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13932.974090] Lustre: Skipped 1 previous similar message [13933.032817] Lustre: tst250fs-MDT0001: new disk, initializing [13933.179435] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13933.239556] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13933.256478] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13937.583377] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [13942.085752] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13949.394484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13949.865179] Lustre: tst250fs-OST0000: new disk, initializing [13949.878700] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13949.898508] Lustre: 375843:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13950.063749] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13951.663443] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13951.689503] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13951.820624] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13957.908073] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [13969.401599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13969.603623] Lustre: tst250fs-OST0001: new disk, initializing [13969.607482] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13969.614692] Lustre: 376714:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13969.728508] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13976.581863] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [13976.662464] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13976.685237] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13976.797943] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13982.299068] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14004.713730] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14004.722160] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14004.742632] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14007.783875] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14007.788664] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14009.372244] Lustre: server umount tst250fs-MDT0000 complete [14013.171872] LustreError: 374205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587988 with bad export cookie 1868332279190584523 [14013.174442] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14013.182329] LustreError: 374205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14013.687407] Lustre: server umount tst250fs-MDT0001 complete [14027.461231] Lustre: server umount tst250fs-OST0000 complete [14041.452224] Lustre: server umount tst250fs-OST0001 complete [14064.498351] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [14073.880819] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [14110.385397] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:14:43 (1787588083) [14112.191167] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14130.991392] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [14141.688141] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [14170.418776] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13915 sec ========= 12:15:43 (1787588143) [14172.075879] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:15:45 (1787588145) === [14175.076559] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:15:48 (1787588148) === [14213.543124] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [14216.453505] Key type lgssc unregistered [14216.756790] LNet: 385846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14216.760523] LNetError: 385846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14217.836770] LNet: Removed LNI 192.168.202.116@tcp [14218.707345] Key type .llcrypt unregistered [14218.710404] Key type ._llcrypt unregistered